RelianceScope: An Analytical Framework for Examining Students' Reliance on Generative AI Chatbots in Problem Solving
Paper
DatasetABSTRACT
Generative AI chatbots enable personalized problem-solving, but effective learning requires students to self-regulate both how they seek help and how they use AI-generated responses. Considering engagement modes across these two actions reveals nuanced reliance patterns: for example, a student may actively engage in help-seeking by clearly specifying areas of need, yet engage passively in response-use by copying AI outputs, or vice versa. However, existing research lacks systematic tools for jointly capturing engagement across help-seeking and response-use, limiting the analysis of such reliance behaviors. We introduce RelianceScope, an analytical framework that characterizes students' reliance on chatbots during problem-solving. RelianceScope (1) operationalizes reliance into nine patterns based on combinations of engagement modes in help-seeking and response-use, and (2) situates these patterns within a knowledge-context lens that accounts for students' prior knowledge and the instructional significance of knowledge components. Rather than prescribing optimal AI use, the framework enables fine-grained analysis of reliance in open-ended student-AI interactions. As an illustrative application, we applied RelianceScope to analyze chat and code-edit logs from 79 college students in a web programming course. Results show that active help-seeking is associated with active response-use, whereas reliance patterns remain similar across knowledge mastery levels. Students often struggled to articulate their knowledge gaps and to adapt AI responses. Using our annotated dataset as a benchmark, we further demonstrate that large language models can reliably detect reliance during help-seeking and response-use. We conclude by discussing the implications of RelianceScope and the design guidelines for AI-supported educational systems.
DEFINING RELIANCE
RelianceScope models student-chatbot interaction as cycles of two actions: help-seeking (asking the chatbot) and response-use (using the chatbot’s reply). Each action is classified by cognitive engagement as passive, active, or constructive. We define reliance as the combination of engagement modes across these two actions.
DATA COLLECTION
We collected a dataset of 79 college students' interactions with a chatbot while solving a web programming problem. Students used the interface shown below to create a to-do list application using Vue.js, with access to a chatbot for help. The dataset includes 1,362 chat logs annotated with reliance patterns, 2,708 code edit logs, pre- and post-assessments, and self-reported self-regulation measures.
RELIANCE PATTERNS
We identified the nine reliance patterns in our dataset and analyzed their distribution and characteristics. Each pattern reflects a unique combination of engagement modes, revealing diverse ways students interact with AI chatbots during problem-solving. The distribution of these patterns are noted in each square below. CLICK on each square to see example conversations from our dataset.
[Constructive Help-Seeking] The student chose to ask for an example.
It looks like you're ready to set up the app's data! To declare the objects for your app, you can return an object from the data() function that includes:
- An array for the list of todos (initially empty or with a sample todo).
- A string for the new todo text (initially empty).
Here's a simple structure you can use:
data() {
return {
todos: [], // This will hold your list of todos
newTodo: "" // This will hold the text for the new todo
};
}
Try adding this to your data() function. Let me know if you need more help!
[Passive Response-Use] The student copied the code verbatim.data() {
return {
todos: [], // This will hold your list of todos
newTodo: "" // This will hold the text for the new todo
};
}
Check more findings and insights in our paper and dataset. We also provide design guidelines for AI-supported educational systems based on our findings.
BIBTEX
