By John W. Verity

It's arguably the No. 1 rule of business: If you can't measure it, you can't improve it. Or as many a CIO has had to learn the hard way, if you can't measure it, you can't communicate its value.

And that lack of clear, forthright communications can have dire consequences: You might find it much harder to get your next development project funded the way you need. Or worse, you might see a major chunk of IT activity handed over to an outsourcer that can back up its business case with reams of metrics, which may be solid or not.

How many IT departments can make a good case about their effectiveness, about what they actually contribute to their business's bottom line? How many CIOs can provide defensible metrics that are credible to non technical line managers and others? Too few, experts say, especially given the mounting pressures they're experiencing from tight-fisted CFOs and aggressive outsourcers.

"Definitely, there's increasing pressure on the IT function to prove that they're adding value to the business," says Gary Curtis,managing director of the strategic IT effectiveness practice at Accenture. "There's very healthy skepticism[in some companies] that IT is not doing that. Therefore, there's more pressure to do a better job with metrics."

IT metrics are hardly new, but the understanding of how to learn from and leverage them is steadily evolving, and their importance is growing as corporations weave information technology into virtually all of their activities. From financial services to manufacturing to retailing, IT is a vital, even strategic lever. Equally important, of course, is that IT budgets are receiving more scrutiny than ever.

In the past, the metrics that IT sought to collect and interpret described mainly technical operations—server utilization, network uptime, number of help desk queries resolved within one hour, and so forth.

Those metrics were easy to collect because the systems involved spewed out voluminous log files and usage stats. Since the 1980s, tremendous effort has gone into measuring and comparing the productivity of different software development methods— another specialized body of metrics.

IT Justification
Now, yet another category of metrics is gaining attention, namely metrics that aim to help IT in better justifying itself at a business level. That means fewer bits and bytes and more dollars and cents. It means calculating fully costed ROIs and being able to associate measurable gains in revenues or market share, for instance, to specific systems and development projects. Showing IT’s value to the overall business is “the holy grail, ”Curtis says. But measuring the effectiveness of IT spending with solid business case methods is still “uncommon to find,” he says.

Indeed, demonstrating the business value of IT is easier said than done, Curtis warns. “The real challenge, more than anything, is that there are no standard metrics concerning business payback.” As cliché as it sounds, IT managers and business managers still tend to view the world in different terms. For example, even when IT reports that a system’s uptime has reached the proverbial “five nines” level, or 99.999 percent, the few minutes that it is down each month may actually cost business managers hours of extra work in calling disgruntled customers and keying in data that had to be captured with paper and pencil.

Moreover, when a new system fails to boost sales, cut costs or improve customer satisfaction—three classic, intended outcomes— it may be quite difficult to figure out what went wrong. Were the system’s requirements not worked out properly? Was its engineering faulty? Were users not trained in the right way? Collecting the right metrics, however, can help to peel away misperceptions and biases and help managers get to the truth of what went wrong. That way, they can better avoid repeating their mistakes.

“Business-centric metrics can lead to a quantitative understanding,” says David Hurwitz, vice president for CA’s Business Service Optimization business unit. “They can help to answer the question, ‘Is there something IT should be doing differently?’“

Curtis’ and others’ advice: Get IT and each business sponsor to agree on exactly how to measure the success of a particular system before it’s actually built or implemented— before any serious money is spent, that is. Whether the system is intended to improve revenues, keep costs in check or improve service, build a solid business case up front, he says.

Take, for example, a new sales support system. “The best practice is, at the beginning of that outlay, to build a relationship between the IT team and the sales-force management so that they can all agree on the business case and how success will be measured in business terms. This measurement might be increased sales calls, or a higher close rate per call, or deltas in revenue per salesperson over a certain period of time—six months or a year,” Curtis says.

“The important thing for IT,” Curtis says, “is to come up with a set of business metrics that the sales force understands.” Unfortunately, he adds, this kind of thinking remains scarce in Corporate America.

Strategic Support
Well-chosen metrics have certainly helped EarthLink, one of the nation’s largest Internet service providers and Web hosters, to justify and better understand a major addition to its customer support tools. The Atlanta-based company has always treated technical support as strategically important, as an opportunity to distinguish itself from other ISPs and to build customer loyalty. And like most companies running call centers, EarthLink is always looking for new ways of resolving customers’ problems faster so as to minimize lengthy, telephone based support sessions.

Recently, the ISP and Web hosting company learned of CA Support Bridge™, a solution that enables support reps to engage in live chats with customers, remotely diagnose customer computer issues and push software scripts to those PCs as away to perform fixes.

Dave Flammia, EarthLink’s director of call center innovation, recalls that he and his team were eager to deploy the CA solution, but he knew he would have to present his case to the company CFO. “We’re not part of the IT group,” Flammia says. “We don’t work for the CIO. We ask for technology innovations and money to buy tools, but any purchases we [want to]make require the CFO’s sign-off.”

In addition to its technical features, CA’s solution appealed to Flammia’s team because of the easy implementation. After putting together a solid business case and a pilot installation, EarthLink implemented the CA Support Bridge solution at EarthLink call centers located in the U.S. and India. “We segued very efficiently, with no downtime,” Flammia says.

Best of all, early surveys revealed a 10 percent improvement in customer satisfaction and a 10 percent uptick in first-call resolution— enough, certainly, to justify the initial investment. And that’s even before the new support tools have been rolled out to all of EarthLink’s call centers worldwide. And because the tools can report on their own activities, Flammia and his team have the metrics they need to prepare any reports that EarthLink’s IT or finance department might request.

In EarthLink’s case, metrics have helped to prove the business case with quantifiable results .But where metrics really get put to the test is when the CFO, for instance, begins to consider outsourcing a major portion of the IT department’s activities. Outsourcers, says Accenture’s Curtis, live and breathe metrics. “They will bring a measure of service and service levels and cost to the table on Day One,” he says, and the savvy CIO will be able to present a comparable set of metrics to argue his or her side of the story.

According to Accenture, which has run surveys on the subject, only around 50 percent of large U.S. companies are regularly building business cases for their IT projects. “The reasons vary by the history and culture of the company,” Curtis says. “One reason is that many companies find it quite difficult to put business-leader types and the relevant IT project leaders together in a sufficiently concentrated way so that they can think together. Everyone’s schedule is full just trying to stay afloat. More often than not, the job of working out the business case is just thrown to IT.”

Curtis sketches out some best practices. The very best IT departments, he says, “know the cost of everything they deliver, and they know it in two dimensions.” First, they know what it costs them to provide their IT services—the goods and services and the labor involved. In addition, Curtis says, they know the cost of delivering their services in terms of business transactions— the cost of the IT services that are needed to, say, open a new retail bank account. “This isn’t easy,” Curtis says, “but the rewards are enormous because it helps you to sort out what to invest in.”

Next, IT departments must understand the key success factors that affect their own, peculiar development processes. It’s well understood that changing key staff or adding developers during the later stages of a project, for instance, will raise costs and defect rates. But failing to keep track of these factors will have a significant impact on risk. Curtis suggests that IT managers maintain a dashboard that collects and presents these key underlying metrics.

All Q&A processes need to be driven by defect rates —“a crucial metric,” Curtis says. Defect rates can help “pinpoint fundamental issues: Is the development team weak? Is the manager weak? Does this technology produce more defects than that technology?”

Curtis says that as soon as possible, in the advanced test or pilot phase of a project, a new system “should be measured according to the business metrics on which its original business case was built in the first place.” If the system was supposed to improve inventory turns at a retail chain, then start measuring those turns as soon as they can be measured.

Finally, Curtis calls for vigilance. Collecting metrics “is not always straight forward,” Curtis says. “It requires careful work. The measurement of business metrics, he says, needs to be made part of the quality assurance function, “woven into the application delivery and test cycle.” Without such discipline, he warns, the gathering of metrics “may get lost in the heat of the battle as a new system gets rolled out. It’s hard work, building metrics into the development methodology.” But the payoff, he says, will be more than worth it.

Case Builders
Michael Mah, senior consultant at Cutter Consortium, an Arlington, Mass., IT advisory firm, points out that developing a business case can help put IT spending in perspective and align it better with the company’s overall product strategy. Offering a purely hypothetical example, he says that when Apple decided to add Microsoft Windows support to its iPod music player, its engineers could have developed a business case that called for spending more than usual on creating the required software. That extra investment could help ensure that the new Windows compatible iPod swept a fast-emerging and potentially huge market. “To build this killer app, you’re going to accept that it will cost more than other development projects,” Mah says. “You need to build it fast, and you need to build it bug-free.”

Likewise, companies weighing in-house development versus offshore work also can benefit from having the right metrics on hand. Mah recalls a certain CIO who had invested heavily for several years in improving his firm’s development with so-called extreme programming techniques. When the prospect of offshoring came up, with seemingly lower costs, the CIO was able to show that his teams could deliver code 30 percent to 50 percent faster than the industry average, and with 50 percent fewer defects. “We proved that the complexity of working offshore slowed the development effort and drove up the defect rate,” Mah says. “That’s the kind of thing that management by numbers can tell you.”

The importance of metrics will only increase as IT departments strive to organize and manage themselves as in-house service providers. By offering fellow business units catalogs of well-defined services— fixing up a new employee with PC, network passwords and an e-mail account, for instance—that carry predetermined prices and are governed by Service Level Agreements (SLAs),  IT can identify inefficiencies and benchmark itself against industry service levels. Service-centric IT management, more importantly, helps managers focus on delivering more business value and avoid distraction from underlying technologies.

Ultimately, Curtis says, the use of IT metrics will only be as good as the people that collect and leverage them. “There’s a fine line between art and science,” he says. “There are lots of estimation methods for projects, but at the end of the day, it’s all about experience in building quality estimates. Having done it a number of times before is the most important thing. It’s something the best IT teams are good at. It’s hard to be 100 percent, and nobody ever is.” But the error rate for experienced teams, he says, can be as little as 10 percent to 15 percent, versus “double or worse” that novices may come up with.

Not surprisingly, he advocates calling On  third parties to work as neutral observers in the creation of business cases. Outsiders, he says, can add a much-needed measure of objectivity and keep the process free of political bias. Too often, Curtis points out, “IT feels pressure to come up with certain numbers, to meet a budget number or match whatever’s left in the kitty.”

Who knows? Maybe the most forward thinking companies will establish the post of chief metrics officer.

John W. Verity writes about technology and business from South Orange, N.J.