TeachTune: Reviewing Pedagogical Agents Against Diverse Student Profiles with Simulated Students
TeachTune: Reviewing Pedagogical Agents Against Diverse Student Profiles with Simulated Students
Hyoungwook JinKAIST
Minju YooEwha Womans University
Jeongeon ParkUC San Diego
Yokyung LeeKAIST
Xu WangUniversity of Michigan
Juho KimKAIST
Hyoungwook Jin
KAIST
Minju Yoo
Ewha Womans University
Jeongeon Park
UC San Diego
Yokyung Lee
KAIST
Xu Wang
University of Michigan
Juho Kim
KAIST
Paper
Slides🚀 If you want to try out TeachTune, please contact jinhw@kaist.ac.kr! 🚀
ABSTRACT
Large language models (LLMs) can empower teachers to build pedagogical conversational agents (PCAs) customized for their students. As students have different prior knowledge and motivation levels, teachers must review the adaptivity of their PCAs to diverse students. Existing chatbot reviewing methods (e.g., direct chat and benchmarks) are either manually intensive for multiple iterations or limited to testing only single-turn interactions. We present TeachTune, where teachers can create simulated students and review PCAs by observing automated chats between PCAs and simulated students. Our technical pipeline instructs an LLM-based student to simulate prescribed knowledge levels and traits, helping teachers explore diverse conversation patterns. Our pipeline could produce simulated students whose behaviors correlate highly to their input knowledge and motivation levels within 5% and 10% accuracy gaps. Thirty science teachers designed PCAs in a between-subjects study, and using TeachTune resulted in a lower task load and higher student profile coverage over a baseline.
PAPER PRESENTATION
KEY INTERFACES
Designing Pedagogical Agents
Teachers use a node-based interface to create the state machine of a PCA. The state machine starts with a root node that consists of the PCA's start message and the behavioral instruction. The state changes to one of the connected nodes depending on a student's response. When the state changes, PCA receives a new instruction described in the node and behave accordingly. Below is an example of a PCA's state machine diagram created by a teacher in our user study.
Creating Student Profiles
Teachers can organize the types of students they review. TeachTune helps teachers externalize and develop their PCA evaluation space with templated student profiles. Our interface provides knowledge components and student trait inventories to help teachers explore different combinations of different knowledge levels and student traits effectively. Below are the student profiles created by a teacher in our user study.
Knowledge Component
Please check the knowledge this student will be acquiring at the beginning of the conversation.
Goal Commitment
Please consider how this student will respond to each question.
I am strongly committed to pursuing this goal.
Strongly disagree
Strongly agree
I think this is a good goal to shoot for.
Strongly disagree
Strongly agree
I am willing to put forth a great deal of effort beyond what I’d normally do to achieve this goal.
Strongly disagree
Strongly agree
Motivation
Please consider how this student will respond to each question.
I keep working on a problem until I understand it.
Strongly disagree
Strongly agree
I try to learn more about something that I don't understand right away so that I will understand it.
Strongly disagree
Strongly agree
When I know I have learned something new, I feel good inside.
Strongly disagree
Strongly agree
Self-efficacy
Please consider how this student will respond to each question.
I believe I am the kind of person who is good at science.
Strongly disagree
Strongly agree
I believe I am the type of person who can do science.
Strongly disagree
Strongly agree
I believe I can learn well in a science course.
Strongly disagree
Strongly agree
Academic Stress
Please consider how this student will respond to each question.
I feel a lot of pressure in my daily studying.
Strongly disagree
Strongly agree
Future education and employment bring me a lot of academic pressure.
Strongly disagree
Strongly agree
I feel that I have disappointed my parents when my test/exam results are poor.
Strongly disagree
Strongly agree
Reviewing Simulated Chats
TeachTune generates mock chats between a PCA and simulated students. Simulated students acquire knowledge throughout the chats, actively ask questions, show indifference, or exhibit passive learning attitudes according to their profiles. Teachers can use the chats to quickly review different PCA designs without manually typing messages. Teachers can iteratively refine their PCA by identifying and re-evaluating corner cases.
Chatbot
Are you ready to review the concepts you learned last time?
Student
Yes, I am ready to review. What concept would you like to review together?
Student
Learned 3 / 6 items
✨ Solids have a regular particle arrangement, are rigid, have a constant shape and volume, and do not flow.
✨ Liquids have a less regular particle arrangement than solids, change shape but have a constant volume, and flow.
Gases have a highly irregular particle arrangement, have neither a constant shape nor volume, flow, and spread out to fill a space.
Substances exist in only one of the three states of matter—solid, liquid, or gas—but can change to a different state depending on temperature or pressure; the change in a substance’s state is called a phase change.
During a phase change, the properties of a substance do not change because the particles that make up the substance do not change.
✨ Even when a substance changes state, the particles that make up the substance and the number of particles do not change, so the mass does not change.
Chatbot
Your readiness is really great! Let’s think about some real-life examples of state changes. For example, the process of ice melting into water or water boiling into steam. What examples can you think of?
Student
Ah, the examples I have thought of are the process of ice melting into water and water boiling into steam. I think these state changes happen frequently in our daily lives.
Student
Learned 3 / 6 items
Solids have a regular particle arrangement, are rigid, have a constant shape and volume, and do not flow.
Liquids have a less regular particle arrangement than solids, change shape but have a constant volume, and flow.
Gases have a highly irregular particle arrangement, have neither a constant shape nor volume, flow, and spread out to fill a space.
Substances exist in only one of the three states of matter—solid, liquid, or gas—but can change to a different state depending on temperature or pressure; the change in a substance’s state is called a phase change.
During a phase change, the properties of a substance do not change because the particles that make up the substance do not change.
Even when a substance changes state, the particles that make up the substance and the number of particles do not change, so the mass does not change.
Chatbot
You really thought well! The process of ice melting and water boiling is a state change that we experience every day. Understanding the change in state is important, even though the properties of the substance do not change. Do you have any other questions related to this?
Student
Yes, I understand the concept of state changes well. If you have any other questions, please let me know!
Student
Learned 5 / 6 items
Solids have a regular particle arrangement, are rigid, have a constant shape and volume, and do not flow.
Liquids have a less regular particle arrangement than solids, change shape but have a constant volume, and flow.
Gases have a highly irregular particle arrangement, have neither a constant shape nor volume, flow, and spread out to fill a space.
✨ Substances exist in only one of the three states of matter—solid, liquid, or gas—but can change to a different state depending on temperature or pressure; the change in a substance’s state is called a phase change.
✨ During a phase change, the properties of a substance do not change because the particles that make up the substance do not change.
Even when a substance changes state, the particles that make up the substance and the number of particles do not change, so the mass does not change.
Chatbot
Great! I am glad that you have a high level of understanding! Next, let’s talk about why mass does not change when a substance undergoes a change in state. The reason mass does not change is because the particles that make up the substance do not change. What do you think about this?
Student
That’s right! The change in state of a substance does not change the mass because the particles that make up the substance do not change. This point seems really important.
Student
Learned 5 / 6 items
Solids have a regular particle arrangement, are rigid, have a constant shape and volume, and do not flow.
Liquids have a less regular particle arrangement than solids, change shape but have a constant volume, and flow.
Gases have a highly irregular particle arrangement, have neither a constant shape nor volume, flow, and spread out to fill a space.
Substances exist in only one of the three states of matter—solid, liquid, or gas—but can change to a different state depending on temperature or pressure; the change in a substance’s state is called a phase change.
During a phase change, the properties of a substance do not change because the particles that make up the substance do not change.
Even when a substance changes state, the particles that make up the substance and the number of particles do not change, so the mass does not change.
Chatbot
You understood it exactly! The fact that mass does not change during a change in state of a substance is a very important concept. Your thinking is really great! Do you have any other topics you would like to learn more about?
Student
For now, I don’t have any other topics that come to mind. It would be good to learn more about changes in state. If you have any other questions, please let me know!
Student
Learned 5 / 6 items
Solids have a regular particle arrangement, are rigid, have a constant shape and volume, and do not flow.
Liquids have a less regular particle arrangement than solids, change shape but have a constant volume, and flow.
Gases have a highly irregular particle arrangement, have neither a constant shape nor volume, flow, and spread out to fill a space.
Substances exist in only one of the three states of matter—solid, liquid, or gas—but can change to a different state depending on temperature or pressure; the change in a substance’s state is called a phase change.
During a phase change, the properties of a substance do not change because the particles that make up the substance do not change.
Even when a substance changes state, the particles that make up the substance and the number of particles do not change, so the mass does not change.
Chatbot
Your interest in changes in state is really cool! Let’s go a little deeper into the types of changes in state. For example, there are terms like melting, freezing, evaporation, and condensation. Do you know what each of these processes means?
Student
I’m sorry, but I’m not sure about those terms. The concept of changes in state is interesting to me. If you have any other questions, please let me know!
Student
Learned 5 / 6 items
Solids have a regular particle arrangement, are rigid, have a constant shape and volume, and do not flow.
Liquids have a less regular particle arrangement than solids, change shape but have a constant volume, and flow.
Gases have a highly irregular particle arrangement, have neither a constant shape nor volume, flow, and spread out to fill a space.
Substances exist in only one of the three states of matter—solid, liquid, or gas—but can change to a different state depending on temperature or pressure; the change in a substance’s state is called a phase change.
During a phase change, the properties of a substance do not change because the particles that make up the substance do not change.
Even when a substance changes state, the particles that make up the substance and the number of particles do not change, so the mass does not change.
Chatbot
Okay, then let me explain it to you briefly!
- Melting: The process of a solid turning into a liquid when it receives heat (e.g., ice melting into water)
- Freezing: The process of a liquid turning into a solid when it loses heat (e.g., water freezing into ice)
- Evaporation: The process of a liquid turning into a gas when it receives heat (e.g., water evaporating into steam)
- Condensation: The process of a gas turning into a liquid when it loses heat (e.g., steam condensing into water)
Did you understand? If you have any other questions, feel free to ask!
🚀 TEACHERS AND EDUCATORS
Our team is looking for deployment partners. If you are a teacher or an educator interested in using TeachTune in classes, please get in touch with the authors. We would like to conduct longitudinal studies to confirm the effectiveness of TeachTune in real-world settings and receive feedback from teachers and educators in the field. Since we can support the cost of running our software (e.g., OpenAI API, server), do not hesitate to contact us and share your circumstances and needs!
