Comparison of Static Tcl Code Validity Testers

In the course of developing TclTutor I've collected a bunch of scripts with various errors that I use to validate changes I make to the code that parses the tclsh error output. It occurred to me that this set of bad code could be useful for characterizing the behavior of the static Tcl code correctness packages that have recently been developed. It then occurred to me that someone else might find this useful.

The bad code snippets I've used are *not* designed to actually put the code validation suites through their paces. Many of these code snippets have several errors, ranging from deliberate errors placed to test the parsers to random typo's that broke things in unexpected ways. I won't claim that this is anything close to an exhaustive test suite, or that it's not biased in some manner.

I'd like to thank the folks who have written these static code checkers. Tcl is an extremely difficult language for static checking, and the level of success that these packages attain is impressive.

This suite is available to developers who would like to use it to test their package, and I'll be happy to update this page when new packages become available.

Also a quick thanks to Wayne Christopher at ICEM for his encouragement in my presenting this data. ICEM was the first to request a copy of the suite to test their package.

Packages Tested

tclsh

This isn't really a static checker, but the error output from the tclsh shell is pretty much the definition of the error. Usually. Sometimes the parser doesn't actually report the true error. Version 7.6 was used in this test.

tclCheck

This package does a minimal check to confirm that braces, brackets, parentheses and quotes are matched and nested appropriately. It can become confused when these items are in comments, or escaped. I used version 1.1.10.

tcl_lint

This is a full service checker that looks at braces, undefined variables, syntax of known commands, etc. It's a commercial offering from ICEM, but evaluation copies are available for selected OS's.

tclparse

This is a full service checker that looks at braces, undefined variables, syntax of known commands, etc. It's a very ambitious student project, available in source code. Version 2.0 was used in this test.

tclparse 3.0

This is a full service checker that looks at braces, undefined variables, syntax of known commands, etc. It's a very ambitious student project, available in source code. Version 3.0 beta was used in this test.

tcl_cruncher

This tool is designed primarily as a Tcl script reformatter to make scripts execute faster. It does some syntax checking as well. The version tested is 1.11.

Executive Summary

Results

Package	Errors	No Error	Error
	Identified	Detected	Misidentified
tclsh	15	0	2
tclcheck	8	9	0
tcl_lint	8	4	5
tclparse	6	1	10*
tclparse 3.0	13	1	3
tcl_cruncher	6	9	2

*TclParse sometimes identified a real error after the error which caused the failure. Since the execution halting error was missed, I counted this as a misdiagnosis.

Long Form