Jul 10th, 2007 | Free Software, Tools, Software development | 2 Comments
After my last post I thought I should look a little deeper into code metrics. Unsurprisingly, a lot has been done in this area- researchers have been investigating metrics since at least the mid-70s. I’m not sure how active the field is today.
There are numerous commercial offerings of tools that will generate metrics for a codebase, but relatively few open source ones, at least for C and C++. Presumably this is because of the difficulty of developing a parser for the tortured syntax of C++. The best open-source tool I found was cccc which unfortunately is no longer under active development. cccc was written by Tim Littlefair for his PhD at Edith Cowan University in Perth, making it home-grown open source. Cool! It uses PCCTS (The Purdue Compiler-Compiler Tool Set) as a parser and generates XML and HTML files containing the calculated metrics.
The range of metrics calculated is good, although the HTML output is fairly basic (sorry Tim), and there’s no graphs. I ran cccc over my pet project Springysim, the resulting output is here.
The metrics produced by cccc are divided into three groups: procedural, object-oriented and structural:
Procedural metrics include Lines of Code (LOC), Lines of Comment (COM), McCabe’s cyclomatic complexity measure and various ratios of these numbers. The concept of cyclomatic complexity was introduced by McCabe in his 1976 paper and the cccc documentation has this to say about it:
The formal definition of cyclomatic complexity is that it is the count of linearly independent paths through a flow of control graph derived from a subprogram. A pragmatic approximation to this can be found by counting language keywords and operators which introduce extra decision outcomes. This can be shown to be quite accurate in most cases. In the case of C++, the count is incremented for each of the following tokens: ‘if’,'while’,'for’,’switch’,'break’,'&&’,'||’
This intuitively seems like a useful metric, although I’d like to read some studies validating it in practice.
Objec-oriented metrics produced by cccc for each class include:
- Weighted methods per class (WMC). In the simplest case the weighting of each method is just one.
cccc also provides WMCv, which only counts public and protected methods.
- Depth of inheritance tree (DIT)
- Number of children (NOC)
- Coupling between objects (CBO). This is the number of other classes that are coupled to a class either as clients or a suppliers.
All these metrics were originally proposed by Chindamber and Kemerer in their 1994 paper A Metrics Suite for Object Oriented Design. It’s not a bad read, but does spend quite some time proving that the proposed metrics satisfy various formal properties proposed by Weyuker in her 1988 paper Evaluating Software Complexity Measures; these parts might be a little dry for some. But it’s not all ivory tower stuff, they also evaluated the metrics by collecting empirical samples at two different software development organisations. However, no attempt was made to correlate the code metrics with project outcomes such as defect rates or maintenance costs.
Unfortunately cccc does not calculate the 5th and 6th metrics suggested by Chindamber and Kemerer. The 6th metric, Lack of Cohesion in Methods (LCOM), examines which instance variables are used by which methods of a class. A class with a single instance variable that is used by all methods has high cohesion, while a class with many instance variables each used by few methods will have a low cohesion. This seems like an interesting metric for OO designers to know.
The structural metrics calculated by cccc are:
- Fan-in: The number of other modules that pass information into a module.
- Fan-out: The number of other modules that a module passes information to.
- An “Information Flow measure” calculated as the square of the product of the fan-in and fan-out of a single module.
These metrics were proposed by Henry and Kafura in their 1981 paper Software Structure Metrics based on Information Flow, this unfortunately does not seem to be freely available. This paper is super-cool as the code base they use for evaluating the metrics is UNIX, version 6. The Lions book is cited as a reference- even cooler!
Tragic fawning over old-school UNIX aside, the paper shows that the information flow measure described above is strongly correlated with the occurrence of changes in the UNIX sources. That is, modules with a high value of the metric also had many changes made to them. The number of changes in a module is used as a proxy for the number of errors in a module, on the assumption that these two measures are strongly correlated.
cccc looks like an interesting tool, or at least the beginning of one. To be useful during development, it would be nice to see how these metrics are changing over time, and cccc doesn’t provide any facilities for that.
Jun 8th, 2007 | Software development | 16 Comments
Earlier in the week I was reading an old post by my new favourite blogger Dennis Forbes entitled Internal Code Reuse Considered Dangerous. It contained this gem:
The question every organization needs to ask itself, then, is what value they could sell their “reusable code” for - what, realistically, would competitors and new entrants in the field offer for it? The answer, in almost every case, is $0, and they wouldn’t want it even at that price. There is extraordinarily little code theft in this industry (even though we’re in the era of burnable DVDs and USB keys) because most code - above and beyond the industry-wide frameworks and libraries - has no value at all outside of a specific project with a specific group of developers. Trying to use it for other projects is often worse than starting with nothing at all.
This is something that I already sort of knew, but it’s the first time I’ve seen somebody express the idea so starkly. It seems very counterintuitive that something that is so difficult and expensive to create can have so little value. I guess that’s why so many organisations like to pretend that their code is much more valuable than it really is.
I first encountered this phenomenon a couple of years back when I was dispatched over 8,000 km from my beloved workstation to assist a customer with an integration project. They were attempting to integrate our product into their product and had run into a few difficulties. A manager at my company suggested that they send us the partially-completed integration, including their source code, so that our engineers could assist. The customer refused. They were jealously protective of their “intellectual property” and would not let any of it off-site.
Since this customer was a big fish, I was sent to complete the integration at the customer’s premises. I arrived on a Monday morning and was shown to my cube where the hardware I needed was already assembled. I sat down and fired up Vim to start looking at their code and was INSTANTANEOUSLY BLINDED by the reeking bile that was pouring across my monitor. Even if I or my company were interested in marketing a product that did the same thing as the customers product, the last thing we would ever do is steal this code. Incorporating their code into our product would have meant incorporting all their bugs into our product, and from the look of it there were probably a couple for every hundred lines.
Of course, the code was not worthless to the customer- it was in a successful product that was making them money. As Dennis says, it was just closely tied to “a specific project with a specific group of developers”.
May 29th, 2007 | Software development | No Comments
I was going to write a post describing various dodgy methods for tying together version control and issue tracking systems, but I’ve found a better way! Read on for the thrilling details.
In his post on bug tracking techniques, James mentions one reason to refer to issue-tracker Issues in version control commit messages:
The solution I came up with was to ask everyone to put a specially formatted string in the commit log that noted the bug number that was resolved in that commit. The version control system will keep track of the code that gets integrated between branches, and carries the commit messages with it. That means getting a reliable list of bugs fixed in any given version is as simple as enumerating the changesets integrated into that branch.
As described, this is a method for generating a list of changes in a given software release. Being able to look at a source code change and know what Issue the change was made for is another benefit. This helps with those “Why the hell did he do that?” questions that come up when looking at Other People’s Code.
After adding these strings to commit messages for a while, I realized that it’s just as useful to have a mapping the other way - from the issue tracker to the version control system. This allows viewing the changes committed to fix a given bug or implement a feature. I did this by adding a comment to an Issue, e.g.
Fixed on 5.1 branch in Perforce change 12345.
This is really easy with Perforce since the changelist number uniquely identifies a set of changes to multiple files that were atomically committed to the repository. Doing it with CVS is trickier.
This manual method was useful but very primitive. A better method is to use the post-commit hook provided by the version control system to append a message to an Issue. In the example above, the changelist number (and perhaps the commit message) would be appended to the Issue page by a script when a commit is done. This is possible because the Issue number is (manually) included in the commit message.
The best method is to use one of the pre-existing solutions to tie the issue tracker and version control system together. Until recently, I wasn’t aware that there are a bunch of these for Perforce, as listed here. There are plugins for Jira, FogBugz and Bugzilla among others.
For the JIRA/Perforce combination I’ve used previously, a Jira plugin is available from Atlassian and it looks really cool. For each Jira Issue you can see a list of Perforce changes, including the commit message and a list of changed files. Each Perforce change entry can be hyperlinked to the change itself as displayed by p4web.
I think this would also help with James’ original problem of compiling a change list for a particular release. Jira and Perforce are integrated using the Perforce jobs feature, and as described here, jobs are preserved across integrations. So it’s possible to ask Perforce which Issues (jobs) have been fixed on any branch without having to add a specially formatted string to commit messages.
May 16th, 2007 | Software development | No Comments
James over at Code Lore has a post up with some good tips on using bug trackers. This is great news because I have been planning to write a very similar post but have not gotten around to it. Prior to my departure from my previous employer, James was the Program Manager for the product I was working on. There were a few things we were doing that were kinda cool (or at least new to me) and worth blogging about.
First off, I don’t like calling them “bug trackers”. I prefer “issue tracker”. Everything should go in the issue tracker. This is how our team worked at Sensory and I thought it was great. New features, bugs, and any other tasks that have to be done- including things that will not result in code modifications- are all added to the issue tracker.
Having everything recorded in one place is vastly preferable to other methods that I’ve used- say where bugs are in the bug tracker but new features are “tracked” in a Word document lying around on a shared drive in a folder that’s rarely touched except by the guy who updates the document on the odd occasion when he’s had enough coding for the day. Such techniques are variously referred to as “lightweight” and “totally lame”.
And some random comments on what James wrote:
Logging - i.e., using the “comment” feature of the bug tracker to keep track of progress (or lack thereof). Yes. Yes. Yes.
Bug Triage - I didn’t do any of this since I was a mere developer, but I was present when James was on some of his bug triage rampages. They were brutal.
Scheduling - that is, keeping track of the estimated and actual amount of time spent working on an issue. I was quite skeptical when we started doing this, as I thought is would be too much work and too inaccurate to be very worthwhile. Most developer’s days are filled with many different activities, some done concurrently, so it’s often difficult to know how to allocate time to different issues.
I eventually realized that these numbers do not need atomic clock precision to be very useful. I use and recommend Joel Spolsky’s method for working out what time went where:
You do not really have to watch your stopwatch while you code. Right before you go home, or go to sleep under the desk if you’re one of those geeks, pretend you’ve worked for 8 hours (ha!), figure out which tasks you’ve worked on, and sprinkle about 8 hours in the elapsed column accordingly.
Joel is talking about using Excel for scheduling, but the advice applies just as well when using an issue tracker.
James also discussed integrating an issue tracker with a version control system, and I’ve got another rant about that in the pipeline.
May 15th, 2007 | Software development | No Comments
A cool post by Eric Sink on software requirements. I’m currently re-reading Managing Software Requirements, and Eric’s post is much easier going.
You can check out a pre-release of Fortress, SourceGear’s new product for Application Lifecycle Management (ALM). Currently it looks like an issue tracker, but apparently requirements features are in the pipeline. I like the idea of an inexpensive and lightweight tool for this, as I’ve been burned by inadequate requirements before- even on small projects that are being implemented by one or two people. It’s amazing how easy it is to get this wrong.
May 4th, 2007 | Software development | No Comments
I coworker of mine is currently reading J2EE Development without EJB, and today he pointed out a couple of very nice quotes from C.A.R. Hoare that are included in a chapter entitled “The Simplicity Dividend”. The quotes are from Hoare’s speech at the 1980 ACM Annual Conference, where he was presented with the ACM Turing Award for that year.
In the first quote, Hoare relates his reaction to a specification for a new version of ALGOL in 1965:
I gave desperate warnings against the obscurity, the complexity, and overambition of the new design, but my warnings went unheeded. I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies.
The first method is far more difficult. It demands the same skill, devotion, insight, and even inspiration as the discovery of the simple physical laws which underlie the complex phenomena of nature. It also requires a willingness to accept objectives which are limited by physical, logical, and technological constraints, and to accept a compromise when conflicting objectives cannot be met.
In the second quote, Hoare gives his opinion on the lengthy project that eventually produced the PL/I programming language:
At first I hoped that such a technically unsound project would collapse but I soon realized it was doomed to success. Almost anything in software can be implemented, sold, and even used given enough determination. There is nothing a mere scientist can say that will stand against the flood of a hundred million dollars. But there is one quality that cannot be purchased in this way–and that is reliability. The price of reliability is the pursuit of the utmost simplicity. It is a price which the very rich find most hard to pay.
And one last quote, not included in the book, shows that even a career as stellar as Hoare’s had its share of failures:
There was no escape: The entire Elliott 503 Mark II software project had to be abandoned, and with it, over thirty man-years of programming effort, equivalent to nearly one man’s active working life, and I was responsible, both as designer and as manager, for wasting it.
This project spanned several years in the mid-60s. Hoare was doing commercial software development in the days before it was widely appreciated just how challenging software can be, and the whole speech is well worth reading to get the flavor of those times.
May 3rd, 2007 | Tools, Software development | No Comments
Jason has an interesting post up over at a little madness on the new “personal builds” feature in Pulse.
By way of background, I used to work with Jason at Sensory Networks before he departed to found zutubi and work on Pulse full time. We used Pulse at Sensory to do our builds and I liked it- I prefer it to Cruise Control, the other CI tool I’ve used.
The part that got my attention was this:
Personal builds allow individual developers to submit their changes to Pulse for testing before committing to version control. Pulse does a build and test of the latest version with the developer’s changes applied … personal builds also allow other powerful usage patterns, such as using the distributed building features of Pulse to test across multiple platforms before checking in.
Hot damn! This sounds great. At Sensory, we supported multiple Linux distros and it was quite annoying how often a change that worked fine on Debian broke the RHEL build, or vice versa. Of course, all changes broke the Redhat 9 build, it’s part of the license agreement.
At my current employer we also do builds on multiple platforms, and it’s not much more fun that it was at Sensory.
Ideally you would build and test on all platforms before committing a change, but the time to do this is prohibitive- a couple of hours, at least. Being able to automatically test builds on all platforms before checkin would get round the cross-platform build problem.
