wsj20010128 / m-m-s

M&M’s: A Comparative Analysis of LLMs and Mentors in Student Summary Evaluation

Shijie Wang, Xuanzhang Liu, Shan Jiang

Abstract: Utilizing large language models (LLMs) for educational assessments, particularly in summary evaluation, presents both significant potential and substantial challenges. Recent studies, such as GPT Score and G-EVAL, have explored the capability of assessing the quality of texts generated by natural language generation (NLG) systems, demonstrating a high correlation with human evaluations. However, these approaches have not comprehensively examined the performance and potential biases of large language models (LLMs) and human teachers in evaluating human-written content, such as student summaries. In this work, we employed GPT-3.5 Turbo with chain-of-thoughts (CoT) and a form-filling paradigm, to assess the quality of student summaries and conducted a comprehensive comparison with human teacher evaluations. Our findings suggest that LLMs, while capable in some areas, do not yet match the nuanced judgment of human evaluators, particularly in terms of reliability and bias. These results highlight the need for a critical reevaluation of LLMs in educational settings and suggest enhancements to better align their capabilities with human standards.