Using Generative AI and Multi-Agents to Provide Automatic Feedback

When LLMs collaborate, they write significantly better feedback on student work

Guo, S., Latif, E., Zhou, Y., Huang, X., & Zhai, X. (2024). Using Generative AI and Multi-Agents to Provide Automatic Feedback (No. arXiv:2411.07407). arXiv. https://doi.org/10.48550/arXiv.2411.07407

Two “heads” are better than one, even when it comes to having LLMs write feedback on student work.

The study developed a multi-agent system consisting of two AI agents: one for generating feedback and another for validating and refining it. The system was tested on a dataset of 240 student responses, and its performance was compared to that of a single-agent LLM. Results showed that AutoFeedback significantly reduced the occurrence of over-praise and over-inference errors, providing more accurate and pedagogically sound feedback. The findings suggest that multi-agent systems can offer a more reliable solution for generating automated feedback in educational settings, highlighting their potential for scalable and personalized learning support.

There’s lots of promise in this approach, but there’s still work to do.

[W]e have to admit that even when using multi-agents for generating automatic feedback, some responses still suffer from issues of over-praise and over-inference. In some cases, these issues were detected but not fully revised. Sometimes Agent 2 can provide an incorrect evaluation of the feedback, leading to unsatisfactory revisions.