Teaching with AI: A Systematic Review of Chatbots, Generative Tools, and Tutoring Systems in Programming Education
Keywords:
AI agents; Programming education; Computer science educationAbstract
This review examines the role of artificial intelligence (AI) agents in programming education, focusing on how these tools are being integrated into educational practice and their impact on student learning outcomes. An analysis of 58 peer-reviewed studies published between 2022 and 2025 identified three primary categories of AI agents: chatbots, generative AI (GenAI), and intelligent tutoring systems (ITS), with GenAI being the most frequently studied. The studies report that the primary instructional objectives include providing enhanced programming support 94.83% of studies, delivering motivational and emotional benefits 18.96%, and increasing efficiency for educators 6.90%. Reported benefits include personalized feedback, improved learning outcomes, and time savings. The review also highlights challenges, such as implementation barriers documented in 93.10% of studies, overreliance resulting in superficial learning in 65.52%, and concerns regarding AI errors and academic integrity. These findings suggest the need for instructional frameworks that prioritize the development of prompt engineering skills and human oversight to address these issues. This review provides educators and curriculum designers with an evidence-based foundation for the practical and ethical integration of AI in programming education.
https://doi.org/10.26803/ijlter.25.1.1
References
Akçapnar, G., & Sidan, E. (2024). AI chatbots in programming education: guiding success or encouraging plagiarism. Discover Artificial Intelligence, 4(1), 87. https://doi.org/10.1007/s44163-024-00203-7
Alpizar-Chacon, I., & Keuning, H. (2025). Student’s use of generative AI as a support tool in an advanced web development course. In Proceedings of the 30th ACM Conference on Innovation and Technology in Computer Science Education. V. 1, (pp. 312–318).ACM. https://doi.org/10.48550/arXiv.2503.15684
Alyoshyna, Y. (2024). AI in programming education: Automated feedback systems for personalized learning. [Master's thesis, University of Twente]. https://purl.utwente.nl/essays/101073
Amoozadeh, M., Nam, D., Prol, D., Alfageeh, A., Prather, J., Hilton, M., Srinivasa Ragavan, S., & Alipour, A. (2024). Student-ai interaction: A case study of CS1 students. In Proceedings of the 24th Koli Calling International Conference on Computing Education Research, (pp. 1–13). https://doi.org/10.1145/3699538.3699567
Asgari, M., Tsai, F.-C., Mannila, L., Strömbäck, F., & Sadique, K. M. (2024). Students’ perspectives on using digital tools in programming courses: A cross country case study between Sweden and Taiwan. Discover Education, 3(1), 57. https://doi.org/10.1007/s44217-024-00144-4
Baethge, C., Goldbeck-Wood, S., & Mertens, S. (2019). SANRA-a scale for the quality assessment of narrative review articles. Research Integrity and Peer Review. 2019; 4: 5. https://doi.org/10.1186/s41073-019-0064-8
Bakar, E. E. A., Abd Halim, N. D., Hanid, M. F. A., & Inderawati, R. (2025). The challenges of teaching and learning programming in schools: Insights from a systematic literature review. Karya Journal of Emerging Technologies in Human Services, 1(1), 48–63. https://doi.org/10.37934/kjeths.1.1.4863
Balse, R., Prasad, P., & Warriem, J. M. (2023). Exploring the potential of GPT-4 in automated mentoring for programming courses. In Proceedings of the ACM Conference on Global Computing Education Vol 2, (pp. 191). https://doi.org/10.1145/3617650.3624946
Becker, B. A., Craig, M., Denny, P., Keuning, H., Kiesler, N., Leinonen, J., Luxton-Reilly, A., Prather, J., & Quille, K. (2023). Generative AI in introductory programming. In Computer Science Curricula, (pp. 438–439). https://doi.org/10.1145/3686852.3687073
Bejarano, A., Dickey, E., & Setsma, R. (2025). Implementing the AI-Lab framework: enhancing introductory programming education for CS majors. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 2, (pp. 1383–1384). https://doi.org/10.1145/3641555.3705201
Bien, J., & Mukherjee, G. (2025). Generative AI for data science 101: Coding without learning to code. Journal of Statistics and Data Science Education, 33(2), 129–142. https://doi.org/10.1080/26939169.2024.2432397
Boguslawski, S., Deer, R., & Dawson, M. G. (2025). Programming education and learner motivation in the age of generative AI: student and educator perspectives. Information and Learning Sciences, 126(1/2), 91–109. https://doi.org/10.1108/ILS-10-2023-0163
Bull, C., & Kharrufa, A. (2024). Generative artificial intelligence assistants in software development education: A vision for integrating generative artificial intelligence into educational practice, not instinctively defending against it . IEEE Software , 41(02), 52–59. https://doi.org/10.1109/MS.2023.3300574
Cámara, J., Troya, J., Burgueño, L., & Vallecillo, A. (2023). On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML. Software and Systems Modeling, 22(3), 781–793. https://doi.org/10.1007/s10270-023-01105-5
Chang, C. I., Choi, W. C., & Choi, I. C. (2024a). A systematic literature review of the opportunities and advantages for AIGC (OpenAI ChatGPT, Copilot, Codex) in programming course. In Proceedings of the 2024 7th International Conference on Big Data and Education, (pp. 29–35). https://doi.org/10.1145/3704289.3704301
Chang, C. I., Choi, W. C., & Choi, I. C. (2024b). Challenges and limitations of using artificial intelligence generated content (AIGC) with ChatGPT in programming curriculum: A systematic literature review. In Proceedings of the 2024 7th Artificial Intelligence and Cloud Computing Conference, (pp. 372–378). https://doi.org/10.1145/3719384.3719439
Chen, A., Wei, Y., Le, H., & Zhang, Y. (2024). Learning-by-teaching with ChatGPT: The effect of teachable ChatGPT agent on programming education. British Journal of Educational Technology. https://doi.org/10.1111/bjet.70001
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024). Prompt problems: A new programming exercise for the generative AI era. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, (pp. 296–302). https://doi.org/10.1145/3626252.3630909
Denny, P., MacNeil, S., Savelka, J., Porter, L., & Luxton-Reilly, A. (2024). Desirable characteristics for ai teaching assistants in programming education. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 (pp. 408–414). https://doi.org/10.1145/3649217.3653574
Deriba, F., Sanusi, I. T., O Campbell, O., & Oyelere, S. S. (2024). Computer programming education in the age of generative AI: Insights from empirical research. https://doi.org/10.2139/ssrn.4891302
Dolinsky, M. (2025). Generative artificial intelligence and introductory programming education at universities: Content, assignments, assessment. WSEAS Transactions on Advances in Engineering Education, 22, 41–50. https://doi.org/10.37394/232010.2025.22.5
Fan, G., Liu, D., Zhang, R., & Pan, L. (2025). The impact of AI-assisted pair programming on student motivation, programming anxiety, collaborative learning, and programming performance: a comparative study with traditional pair programming and individual approaches. International Journal of STEM Education, 12(1), 16. https://doi.org/10.1186/s40594-025-00537-3
Fodouop Kouam, A. W. (2024). The effectiveness of intelligent tutoring systems in supporting students with varying levels of programming experience. Discover Education, 3(1), 278. https://doi.org/10.21203/rs.3.rs-4788797/v1
Frankford, E., Sauerwein, C., Bassner, P., Krusche, S., & Breu, R. (2024). AI-tutoring in software engineering education. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training, (pp. 309–319). https://doi.org/10.1145/3639474.3640061
Garcia, M. B. (2025). Teaching and learning computer programming using ChatGPT: A rapid review of literature amid the rise of generative AI technologies. Education and Information Technologies, 1–25. https://doi.org/10.1007/s10639-025-13452-5
Hartley, K., Hayak, M., & Ko, U. H. (2024). Artificial intelligence supporting independent student learning: An evaluative case study of ChatGPT and learning to code. Education Sciences, 14(2), 120. https://doi.org/10.3390/educsci14020120
Ho, C.-L., Liu, X.-Y., Qiu, Y.-W., & Yang, S.-Y. (2024). Research on innovative applications and impacts of using generative AI for user interface design in programming courses. In Proceedings of the 2024 International Conference on Information Technology, Data Science, and Optimization, (pp. 68–72). https://doi.org/10.1145/3658549.3658566
Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M.-C., Vedel, I., & Pluye, P.. (2018). The mixed methods appraisal tool (MMAT) version 2018 for information professionals and researchers. Education for Information, 34(4), 285–291. https://doi.org/10.3233/EFI-180221
Hoq, M., Shi, Y., Leinonen, J., Babalola, D., Lynch, C., Price, T., & Akram, B. (2024). Detecting ChatGPT-generated code submissions in a CS1 course using machine learning models. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, (pp. 526–532). https://doi.org/10.1145/3626252.3630826
Hudin, S. S. (2023). A systematic review of the challenges in teaching programming for primary schools’ students. Online Journal for TVET Practitioners, 8(1), 75–88. https://doi.org/10.30880/ojtp.2023.08.01.008
Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S. H., Isohanni, E., Korhonen, A., Petersen, A., Rivers, K., Rubio, M., Shearad, J., Skupas, B., Spacco, J., Szabo, C., Toll, D. (2015). Educational data mining and learning analytics in programming: Literature review and case studies. In Proceedings of the 2015 ITiCSE on Working Group Reports, (pp. 41–63). https://doi.org/10.1145/2858796.2858798
Jin, H., Lee, S., Shin, H., & Kim, J. (2024). Teach AI how to code: Using large language models as teachable agents for programming education. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, (pp. 1–28). https://doi.org/10.1145/3613904.3642349
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023). Studying the effect of AI code generators on supporting novice learners in introductory programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, (pp. 1–23). https://doi.org/10.1145/3544548.3580919
Keuning, H., Alpizar-Chacon, I., Lykourentzou, I., Beehler, L., Köppe, C., de Jong, I., & Sosnovsky, S. (2024). Students’ perceptions and use of generative AI tools for programming across different computing courses. In Proceedings of the 24th Koli Calling International Conference on Computing Education Research, (pp. 1–12). https://doi.org/10.1145/3699538.3699546
Kiesler, N., & Schiffner, D. (2023). Large language models in introductory programming education: ChatGPT’s performance and implications for assessments. https://doi.org/10.48550/arXiv.2308.08572
Lau, S., & Guo, P. (2023). From" Ban it till we understand it" to" Resistance is futile": How university programming instructors plan to adapt as more students use AI code generation and explanation tools such as ChatGPT and GitHub Copilot. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1, (pp. 106–121). https://doi.org/10.1145/3568813.3600138
Lepp, M., & Kaimre, J. (2025). Does generative AI help in learning programming: Students’ perceptions, reported use and relation to performance. Computers in Human Behavior Reports, 18, 100642. https://doi.org/10.1016/j.chbr.2025.100642
Liu, R., Zenke, C., Liu, C., Holmes, A., Thornton, P., & Malan, D. J. (2024). Teaching CS50 with AI: leveraging generative artificial intelligence in computer science education. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, (pp. 750–756). https://doi.org/10.1145/3626252.3630938
Logacheva, E., Hellas, A., Prather, J., Sarsa, S., & Leinonen, J. (2024). Evaluating contextually personalized programming exercises created with generative AI. In Proceedings of the 2024 ACM Conference on International Computing Education Research-Volume 1, (pp. 95–113). https://doi.org/10.1145/3632620.3671103
Luxton-Reilly, A., Simon, Albluwi, I., Becker, B. A., Giannakos, M., Kumar, A. N., Ott, L., Paterson, J., Scott, M. J., Sheard, J., & Szabo, C. (2018). A review of introductory programming research 2003–2017. In Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education, (pp. 342–343). https://doi.org/10.1145/3197091.3205841
Ma, B., Chen, L., & Konomi, S. (2024). Enhancing programming education with ChatGPT: A case study on student perceptions and interactions in a Python course. International Conference on Artificial Intelligence in Education, 113–126. https://doi.org/10.1007/978-3-031-64315-6_9
Manley, E. D., Urness, T., Migunov, A., & Reza, M. A. (2024). Examining student use of AI in CS1 and CS2. Journal of Computing Sciences in Colleges, 39(6), 41–51. https://doi.org/10.5555/3665464.3665469
Manorat, P., Tuarob, S., & Pongpaichet, S. (2025). Artificial intelligence in computer programming education: A systematic literature review. Computers and Education: Artificial Intelligence, 100403. https://doi.org/10.1016/j.caeai.2025.100403
Medeiros, R. P., Ramalho, G. L., & Falcão, T. P. (2018). A systematic literature review on teaching and learning introductory programming in higher education. IEEE Transactions on Education, 62(2), 77–90. https://doi.org/10.1109/te.2018.2864133
Nagakalyani, G., Chaudhary, S., Apte, V., Ramakrishnan, G., & Tamilselvam, S. (2025). Design and evaluation of an AI-Assisted grading tool for introductory programming assignments: An experience report. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1, (pp. 805–811). https://doi.org/10.1145/3641554.3701913
Nguyen, N., & Nadi, S. (2022). An empirical evaluation of GitHub copilot’s code suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories, (pp. 1–5). https://doi.org/10.1145/3524842.3528470
Oli, P., Banjade, R., Chapagain, J., & Rus, V. (2023). Automated assessment of students’ code comprehension using LLMs. ArXiv Preprint ArXiv:2401.05399. https://doi.org/10.48550/arXiv.2401.05399
Pankiewicz, M., & Baker, R. S. (2024). Navigating compiler errors with AI assistance-a study of GPT hints in an introductory programming course. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 (pp. 94–100). https://doi.org/10.1145/3649217.3653608
Penney, J., Pimentel, J. F., Steinmacher, I., & Gerosa, M. A. (2023). Anticipating user needs: Insights from design fiction on conversational agents for computational thinking. In International Workshop on Chatbot Research and Design, (pp. 204–219). https://doi.org/10.1007/978-3-031-54975-5_12
Pesovski, I., Santos, R., Henriques, R., & Trajkovik, V. (2024). Generative AI for customizable learning experiences. Sustainability, 16(7), 3034. https://doi.org/10.1145/3686852.3687073
Phung, T., P?durean, V.-A., Cambronero, J., Gulwani, S., Kohn, T., Majumdar, R., Singla, A., & Soares, G. (2023). Generative AI for programming education: benchmarking ChatGPT, GPT-4, and human tutors. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 2, (pp. 41–42). https://doi.org/10.1145/3568812.3603476
Qian, Y., & Lehman, J. D. (2017). Students’ misconceptions and other difficulties in introductory programming: A literature review. ACM Transactions on Computing Education (TOCE), 18(1), 1–24. https://doi.org/10.1145/3077618
Rahe, C., & Maalej, W. (2025). How do programming students use generative AI? Proceedings of the ACM on Software Engineering, 2(FSE), 978–1000. https://doi.org/10.1145/3715762
Reigeluth, C. M. (2016). Designing technology for the learner-centered paradigm of education. In Instructional-Design Theories and Models, Volume IV (pp. 287–316). Routledge. https://doi.org/10.4324/9781315795478-21
Sadat Shanto, S., Ahmed, Z., & Jony, A. I. (2024). Generative AI for programming education: Can ChatGPT facilitate the acquisition of fundamental programming skills for novices? In Proceedings of the 3rd International Conference on Computing Advancements, (pp. 685–692). https://doi.org/10.1145/3723178.3723268
Servin, C., Karichev, N. v, & Pagel, M. (2024). Unfolding programming: How to use AI tools in introductory computing courses. In Proceedings of the 25th Annual Conference on Information Technology Education, (pp. 49–55). https://doi.org/10.1145/3686852.3687073
Shihab, M. I. H., Hundhausen, C., Tariq, A., Haque, S., Qiao, Y., & Mulanda, B. (2025). The effects of GitHub Copilot on computing students’ programming effectiveness, efficiency, and processes in brownfield programming tasks. ArXiv Preprint ArXiv:2506.10051. https://doi.org/10.1145/3702652.3744219
Silva, C. A. G. da, Ramos, F. N., de Moraes, R. V., & Santos, E. L. dos. (2024). ChatGPT: Challenges and benefits in software programming for higher education. Sustainability, 16(3), 1245. https://doi.org/10.3390/su16031245
Simaremare, M., Pardede, C., Tampubolon, I., Manurung, P., & Simangunsong, D. (2024). Pair programming in programming courses in the era of generative AI: Students’ perspective. In 2024 31st Asia-Pacific Software Engineering Conference (APSEC), (pp. 507–511). https://doi.org/10.1109/APSEC65559.2024.00069
Stone, I. (2023). Exploring the research gap: Generative AI and learning of python programming among post-primary students. In Proceedings of the 2023 Conference on Human Centered Artificial Intelligence: Education and Practice, (pp. 51). https://doi.org/10.1145/3633083.3633099
Stone, I. (2024). Exploring human-centered approaches in generative AI and introductory programming research: A scoping review. In Proceedings of the 2024 Conference on United Kingdom & Ireland Computing Education Research, (pp. 1–7). https://doi.org/10.1145/3689535.3689553
Stray, V., Moe, N. B., Ganeshan, N., & Kobbenes, S. (2025). Generative AI and developer workflows: How GitHub Copilot and ChatGPT influence solo and pair programming. In Proceedings of the 58th Hawaii International Conference on System Sciences .https://doi.org/10.24251/hicss.2025.883
Sun, D., Boudouaia, A., Zhu, C., & Li, Y. (2024). Would ChatGPT-facilitated programming mode impact college students’ programming behaviors, performances, and perceptions? An empirical study. International Journal of Educational Technology in Higher Education, 21(1), 14. https://doi.org/10.1186/s41239-024-00446-5
Takerngsaksiri, W., Warusavitarne, C., Yaacoub, C., Hou, M. H. K., & Tantithamthavorn, C. (2024). Students’ perspectives on AI code completion: Benefits and challenges. In 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC), (pp. 1606–1611). https://doi.org/10.1109/compsac61105.2024.00252
Thomas, D. R., Lin, J., Gatz, E., Gurung, A., Gupta, S., Norberg, K., Fancsali, S. E., Aleven, V., Branstetter, L., Brunskill, E., & Koedinger, K. R. (2024). Improving student learning with hybrid human-AI tutoring: A three-study quasi-experimental investigation. In ACM International Conference Proceeding Series, (pp. 404–415). https://doi.org/10.1145/3636555.3636896
Tugwell, P., & Tovey, D. (2021). PRISMA 2020. Journal of Clinical Epidemiology (Vol. 134, pp. A5–A6). Elsevier. https://doi.org/10.1016/j.jclinepi.2021.04.008
Wang, H., Wang, C., Chen, Z., Liu, F., Bao, C., & Xu, X. (2025). Impact of AI-agent-supported collaborative learning on the learning outcomes of University programming courses. Education and Information Technologies, 1–33. https://doi.org/10.1007/s10639-025-13487-8
Watson, C., & Li, F. W. B. (2014). Failure rates in introductory programming revisited. In Proceedings of the 2014 Conference on Innovation & Technology in Computer Science Education, (pp. 39–44). https://doi.org/10.1145/2591708.2591749
Wermelinger, M. (2023). Using GitHub Copilot to solve simple programming problems. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, (pp. 172–178). https://doi.org/10.1145/3545945.3569830
Xayriddin, N. (2025). Challenges and solutions in teaching programming: an exploration of global and local perspectives. International Journal of Artificial Intelligence, 1(2), 651–655. https://inlibrary.uz/index.php/ijai/article/view/74343
Xie, J. (2024). Improving introductory java programming education through ChatGPT. Journal of Computing Sciences in Colleges, 40(3), 140–150. https://doi.org/10.5555/3722479.3722526
Yan, W., Nakajima, T., & Sawada, R. (2024). Benefits and challenges of collaboration between students and conversational generative artificial intelligence in programming learning: An empirical case study. Education Sciences, 14(4), 433. https://doi.org/10.3390/educsci14040433
Ye, X., Zhang, W., Zhou, Y., Li, X., & Zhou, Q. (2025). Improving students’ programming performance: an integrated mind mapping and generative AI chatbot learning approach. Humanities and Social Sciences Communications, 12(1), 1–13. https://doi.org/10.1057/s41599-025-04846-4
Zambach, S. (2025). AI-enhanced learning: Comparing outcomes in introductory and advanced programming courses. In Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing, (pp. 104–105). https://doi.org/10.1145/3672608.3707909
Zhai, X. (2023). ChatGPT for next generation science learning. XRDS: Crossroads, The ACM Magazine for Students, 29(3), 42–46. https://doi.org/10.1145/3589649
Zviel-Girshin, R. (2024). The good and bad of AI tools in novice programming education. Education Sciences, 14(10), 1089. https://doi.org/10.3390/educsci14101089
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Said Elnaffar, Farzad Rashidi, Abedallah Zaid Abualkishik
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
All articles published by IJLTER are licensed under a Creative Commons Attribution Non-Commercial No-Derivatives 4.0 International License (CCBY-NC-ND4.0).
