Static Analysis of Programming Assignment Submissions in an Introductory CS Course

Abstract

This paper explores how static analysis errors in Java programming assignment submissions can provide insight into student behavior and performance. CS education research is an expanding field, but the use of static analysis tools has not been a major point of emphasis. These tools examine source code directly without compiling or executing it, detecting style issues defined by programming standards of the language in use, as well as potential bugs and constructs that can be simplified. This paper analyzes the results of three of the most common static analysis tools for Java — Checkstyle, PMD, and FindBugs — on programming assignment submissions in Princeton's introductory CS course. One of our main findings is that the quantity of static analysis errors that a student produces is often an indication of that student's strength, with weaker students producing more static analysis errors on assignments than stronger ones. Here, a student's strength is defined by their final course average. Students who leave more errors on their final attempts at assignments also perform worse, both on exams and in the course as a whole. That being said, there was no clear trend line defining a relationship between a student's total error count and their performance — on assignments, exams, and final average. In terms of error frequency, formatting, documentation, and style issues are by far the most prevalent, but there is a subset of errors that are defined to likely represent bugs. Students who produce more of these errors on their initial attempts at assignments also tend to do perform more poorly in the course.