VanLehn, K. (1991). Two pseudo-student:Applications of machine learning to formative evaluation. In R. Lewis & S. Otsuki (Eds.) Advanced Research on Computers in Education. 17-26. Amsterdam:Elsevier.

The need for on-line evaluation of instructional designs. Ideally, Ihe design of instructional material should include ample testing of preliminary designs so that design errors can be detected and repaired before the instruction is deployed. Such fonnative evaluations are rarer than they should be for several reasons:

• Although the final polishing can be left off, all the little design details must be fully worked out before the instruction can be given to human students. It would be much better if fonnative evaluations could be conducted while the design was still in a rough, preliminary state. When design errors are detected early in the design cycle, less time is wasted worldng out the details of designs that will ultimately be discarded.
• Testing instruction with human students can be difficult, time consuming and expensive. Allhough increasing the quality of instruction is always a desirable goal in principle, the time and money needed for fonnative evaluation may not always be justifiable.
• Current testing methods often yield only two kinds of infonnation, neither of which is particularly helpful to designers. First, there is a overall assessment of instructional effectiveness, such as a difference between pre-test and pOst-teSt scores. Second, there are the panicipants' comments as to which parts of the instruction seem to work well, and which did not.

In contrast to the situation in instructional design, consider the state of the art in mechanical and electrical design. After drafting blueprint or schematic of their design on a CAD station, designers can submit it to a simulation program for evaluation. Simulation programs provide very detailed reports on the design's performance which allow the engineer to locate design errors early in the design process. They may not find all the errors, because the simulation programs and testing regimes are not always perfect, but catching errors early is much cheaper than catching them later or not catching them at all.

For a pdf of the chapter, click here .