Decision theoretic selection of tutoring actions: The DT Tutor project

The DT Tutor project explored a decision-theoretic approach for selecting tutorial discourse actions. DT Tutor used a dynamic decision network to probabilistically look ahead to anticipate how its tutorial actions will influence the student and other aspects of the tutorial state. It weighed its preferences regarding multiple competing objectives by the probabilities that they will occur and then selected the tutorial action with maximum expected utility. DT Tutor considered a rich set of tutorial state attributes in order to approach some of the sensitivity and subtlety exhibited by human tutors. These attributes included the discourse state, progress on the tutorial task (e.g., solving problems), and the student's knowledge, focus of attention, and affective state.

The first application of DT Tutor was RTDT (Reading Tutor, Decision-Theoretic), which tested the feasibility of adding a decision-theoretic action selection engine to an existing tutoring system, Project LISTEN's Reading Tutor. The Reading Tutor listened to children as they read aloud and provided coaching using speech and GUI actions. RTDT was not integrated with the main version Reading Tutor, but a prototyped demonstrated that such an integration was feasible.

As a more ambitious application of the technology, DT was integrated into a calculus related rates tutor, a decision-theoretic intelligent tutoring system for teaching calculus related-rates problems. Using data from dozens of calculus students, this tutoring system was evaluated by a panel of experts.

The evaluation was conducted in two phases. First, logs were recorded from interactions of students with a Random Tutor that was identical to DT Tutor except that it selected randomly from relevant tutorial actions. The logs were used to learn many of DT Tutor’s key probabilities for its model of the tutorial state. Second, the logs were replayed to record the actions that DT Tutor and a Fixed-Policy Tutor would select for a large sample of scenarios. The Fixed-Policy Tutor was identical to DT Tutor except that it selected tutorial actions by emulating the fixed policies of the Cognitive Tutors (http://www.carnegielearning.com), which are theoretically based, widely used, and highly effective. The possible action selections for each scenario were rated by a panel of judges who were skilled human tutors. The main hypotheses were that DT Tutor’s action selections would be rated higher than the Fixed-Policy Tutors and higher than the Random Tutor's. This was the first comparison of a decision-theoretic tutor with a non-trivial competitor.

DT Tutor was rated higher than the Fixed-Policy Tutor overall. For all subsets of scenarios except help requests, DT Tutor was rated higher than the Fixed-Policy Tutor. For help requests, the two tutors tied. DT Tutor was also rated much higher than the Random Tutor.

The DT Tutor project was the Ph. D. project of R. Charles Murray. It began around 1997 and ended in August 2005.

Personnel

Sponsors

Publications

Last update: May 14, 2009.