Students’ programming skills into shape with proper feedback
12.09.2012
In programming education exercises are often assessed by automatic feedback. In his dissertation for Aalto University Department of Computer Science and Engineering Otto Seppälä has studied how different assessment methods and tools can convey information about students’ programming skills and how students can best benefit from feedback.
Seppälä asserts that assessment methods should help students to recognise and comprehend their errors. Students may unnecessarily repeat mistakes because automatic feedback does not always reveal how and why the errors come about.
– Mistakes result from various reasons. Getting to the root of them requires a cross-disciplinary approach combining computer science, pedagogy and cognitive psychology, explains Seppälä.
Tracing good code
A crucial skill in writing new code or in recognising programming errors is code tracing. A programmer needs to have the ability to picture the code being constructed as a finished and a fully functioning programme. The continuous mental testing of the programme will bring most of the possible errors to the fore.
Tracing tasks test both programming skills and algorithmic savvy. They also reveal the misconceptions behind errors. Seppälä uses a cooking analogy to clarify the issue.
– Think of a recipe as a programme a cook needs to execute: a series of consecutive actions and often repetition. Lasagne is a prime example because it carries in it the steps of its own making. The end result shows whether the cook has traced the recipe correctly or made mistakes. Misconceptions manifest themselves in the lasagne as different structural errors, describes Seppälä.
Seppälä and his colleagues discovered to their surprise that students with several programming courses under their belt were still often quite inept in tracing code execution. Their research also shows that working with scratch paper had a positive correlation with successful tracing.
Visual algorithm simulation is a method that resembles code tracing. With it students can practice the functioning of different algorithms without having to write them from scratch. Seppälä showed that by emulating the errors students make, carelessness and recurring systematic errors can often be told apart.
– Next we will begin to develop ways to offer students instructive feedback based on our results.
Mutation to reveal poor testing skills
Testing is a significant part of programming skills, and it is taught from first year courses on. It also requires comprehension of code execution and dead-ends in programmes.
Many tests may however give a false impression of a flawless programme. A common testing measure is known as ‘code coverage’. It tells how big a part of the code is executed by the test suite, and lasagne will pinpoint its shortcomings as well.
– A poor test goes through the majority of the lines of code, but does not necessarily go into the functioning or purpose of the programme at all. If the quality of a lasagne were to be measured only by its height, the test would assume the recipe to have been executed perfectly as long as the lasagne was of required height. It can still be charred black or consist of ground beef alone, Seppälä explains.
Seppälä researched if the weaknesses of the tests and the harmful testing habits of students could be corrected with mutation testing. It entails planting random errors in the code to be tested. If the programme still passes the test with its source code tampered – as a lasagne with rice instead of cheese in it – the test is inadequate. By demonstrating that faulty programmes pass poor tests, the limitations of the tests may unravel for the students.
– The results of mutation testing may be difficult for students to understand, so it is still rather unfit for automatic feedback. Teachers are needed to interpret the results for students. How mutation testing could be put into purely automatic use for students is a necessary topic for further research, asserts Seppälä.
Further information:
Otto Seppälä
otto.seppala [at] aalto [dot] fi
tel. +358 50 408 0539
Department of Computer Science and Engineering
Aalto University School of Science
