automaTA: Human-Machine Interaction for Answering Context-Specific Questions

Abstract

When online learners have questions that are related to a specific task, they often use Q&A boards instead of web search because they are looking for context-specific answers. While lecturers, teaching assistants, and other learners can provide context-specific answers on the Q&A boards, there is often a high response latency which can impede their learning. We present automaTA, a prototype that suggests context-specific answers to online learners’ questions by capturing the context of the questions. Our solution is to automate the response generation with a human-machine mixed approach, where humans generate high-quality answers, and the human-generated responses are used to train an automated algorithm to provide context-specific answers. automaTA adopts this approach as a prototype in which it generates automated answers for function-related questions in an online programming course. We conduct two user studies with undergraduate and graduate students with little or no experience with Python and found the potential that automaTA can automatically provide answers to context-specific questions without a human instructor, at scale.

Authors

Changyoon Lee's profile image
Changyoon Lee *
KAIST
Donghoon Han's profile image
Donghoon Han *
KAIST
Hyoungwook Jin's profile image
Hyoungwook Jin *
KAIST
Alice Oh's profile image
Alice Oh
KAIST

Interface

System interface for asking and receiving the answers for function related questions. A learner can type a question on a search bar at the top and receive three function suggestions along with usage examples. If the suggestions are not satisfactory, the learner can ask the question to human teaching assistants.
automaTA is a Google Chrome extension that allows students to ask questions and receive automated answers on their programming task window. automaTA adds to the rich features built in Elice, an online Python learning platform.

Function Suggestions

Learners ask for function suggestions by describing the function they are looking for. automaTA suggests three relevant functions for the description. Learners can click the ‘Useful’ buttons if the suggestion is correct and helpful.

Relevant Code Examples

automaTA provides the code examples from the official Python documentation and peer code. The code example from peer codes can be more helpful for doing programming tasks because they show how functions are used in the tasks.

Ask to Teaching Assistants

Learners can ask questions directly to teaching assistants if the automated answers are unsatisfactory. The answers to the questions may not be provided immediately as teaching assistants answer them manually.

Human-Machine Collaboration

A diagram that explains how human and machine collaboration happens to automate question and answering: 1) apture the context of questions from learners’ codes., 2) handover questions with unsatisfied answer to experts for qualified answers, and 3) train machines with the experts’ answers for automation.
automaTA provides satisfactory answers by turn-taking between humans and machines. In the case when the automated answers by automaTA are not relevant for a question, learners can ask the same question directly to a human teaching assistant. This workflow ensures that learners can get satisfactory help even when automaTA is not mature enough to give useful answers. When a learner asks a question to a teaching assistant, learner’s code and task information are sent to the assistant along with the question. The teaching assistant uses an independent interface to answer the question. The teaching assistant suggests a list of functions and writes an explanation of how those functions can help solve the learner’s task. Question and answer data are then used to improve automaTA to give better answers.