Examining the Accuracy and Consistency of Teacher and ChatGPT Grading EFL Students' Reports: A Comparative Analysis

Artificial intelligence (AI) is a technological field that has been developing at an accelerated pace in recent years: the era of the AI boom brings new discoveries and technical breakthroughs to all domains, including the academic sector. One of the innovations driven by AI in the field of education is the automated essay scoring (AES) system, which allows for the automatic evaluation of students’ written assignments, as well as the provision of feedback and suggestions on how to improve the text. Automatic assessment systems have the potential to play a significant role in the teaching of foreign languages, as written assignments form a substantial part of the academic workload for students in these educational programs. However, like any other technology, AES requires a thorough and comprehensive analysis. This study aims to examine the accuracy and consistency of grading English as a foreign language (EFL) students’ reports, performed by a teacher and by AI tools. The relevance of the present study is dictated by the active and comprehensive development of artificial intelligence technologies: although AES may potentially reduce the workload of teachers, its implementation requires careful consideration, as error correction and accurate assessment of written work are essential for EFL students. The research design includes conducting a comparison experiment on grades and feedback from teachers and two AI tools that can be used as AES systems: ChatGPT and Essaygrader.ai. The methodology of this research is a mixed-method approach that includes both qualitative and quantitative components: statistical data analysis and comparative analysis with the results demonstrated in the form of a statistical table showing the comparison of grades assigned by the AI and the teacher for the same essays for the quantitative part and explicit comparison of the detailed feedback from the teacher and the neural networks and determination of the external factors that may have an impact on the experimental results and that influence AI tools and human professors for the qualitative part. The dataset for the research included 40 essays written by students participating in the English as a foreign language (EFL) program within the same university — Higher School of Economics. Every essay from the dataset was graded by the teacher and both neural networks under the examination — ChatGPT and Essaygrader.ai — according to the five criteria set. A quarter of the dataset (10 out of 40 essays) was double-checked to minimise errors and ensure data integrity. Five essays from the dataset were used as a basis for in-depth feedback analysis. Both grades and feedback from the teacher and the neural networks were compared in percentage terms in order to demonstrate the current state of AES systems development. The theoretical outcomes of this research consist in providing an actual representation of the current situation in the field of AES, derived from the conducted testing. The results obtained can be used both by teachers who are interested in studying a new technology for evaluating essays and are ready to try it in practice, and by developers and testers of artificial intelligence systems suitable for automatic evaluation of text works. As for the practical outcomes of the study, they include the summarised table of grades for the evaluation of essays determined by neural networks, which can be used for further research, such as diachronic comparisons aimed at monitoring the evolution of automatic assessment systems, or synchronous comparative analyses, where the results of this study can be contrasted with data from other AES systems and/or neural networks used as such.