By Kurt Milne
Winter 2007

IT audit and control-related activities are more than merely a necessary cost that IT organizations must incur to meet regulatory compliance. In fact, the effective use of IT control activities found in frameworks such as ITIL® (IT Infrastructure Library), COBIT (Control Objectives for Information and Related Technology), and ISO20000 (formerly BS15000, a service management standard) can actually improve an organization’s operating performance and generate a significant return on investment

That’s the theory the IT Process Institute set out to test. As a nonprofit organization for IT professionals, ITPI assembled a research team that included both practitioners and researchers from Carnegie Mellon University, Florida State University and the University of Oregon. This team created a Web-based survey that was eventually completed by 98 IT organizations of various sizes and geographic regions. Forty percent of the respondents reported having more than 100 people in their IT organization, and nearly 45 percent of respondents described themselves as directors, VPs or C-level managers.

We undertook this study because IT executives have told us they want to see a strong business case for spending on IT audit and IT control activities. That’s not only because they have the responsibility of complying with various regulations that affect IT, but also because they’re responsible for showing a compelling return on all IT investments.

In fact, spending on IT compliance and IT control activities has increased significantly in the last two years, both in the United States and globally, as Sarbanes-Oxley and other privacy and industry-specific regulations have taken effect. IT organizations have responded to these regulations by investing more resources in IT control activities designed to reduce risk through improved process repeatability. But IT operations executives want to know how to measure the positive impact on the overall performance of the IT organization that has resulted from compliance-related spending. That, we hoped, was where our study would come in.

To answer these questions, we asked a broad range of questions designed to support analysis of 63COBITcontrol activities and 25 key operations, security and audit performance measures. We looked at access controls, change controls, release controls, configuration controls, service-level controls and resolution controls. Based on the replies, we identified top, medium and low performers, based on how many of the 25 performance measures they scored in the top 50th percentile of all respondents. We also summarized the performance differences among high, medium and low performers as a way to quantify performance-improvement potential.

IT Works
The good news: Based on analysis of the top performers in this study, our overall conclusion is that IT organizations that focus ongoing audit and control-related resources on what we call foundational control activities will generate a significant return on investment (ROI) realized through improvements in a wide range of key performance measures. Foundational controls are a subset of 21 activities that have the largest impact on the operations, security and audit performance measures.

In fact, organizations that use the greatest number of foundational controls have higher performance than organizations that use the least. And the foundational controls that most differentiate top and medium performers are change and configuration controls.

We assumed that, given limited resources, IT organizations cannot focus on all the best practices found in the ITIL books and 312 COBIT controls with equal vigor. In addition, we assumed that although organizations must implement a broad range of IT controls to manage risk and meet an increasing number of regulatory requirements, a small set of controls have the greatest impact on performance measures. The second step in this analysis was to identify a subset of control activities that have the greatest impact on operations, security and audit performance measures. This ultimately became our list of 21 foundational controls.

We further characterized the controls that differentiate top from medium performers as the activities that sustain and continually improve their control systems. Activities such as enforcing processes and the consistent use of controls to avert high-risk activities proactively stabilize the IT environment. More specifically, the top six controls that differentiate high and medium performers are:

• Do you monitor for unauthorized changes?
• Have you defined consequences for intentional unauthorized changes?
• Do you have a formal process for IT configuration management?
• Do you have an automated process for configuration management?
• Do you track your change success rate?
• Are you able to provide relevant personnel with correct and accurate information on the present IT infrastructure configurations, including their physical and functional specifications?

When comparing medium and low performers, we found the highest differences are spread across various control areas, including release, service level, resolution and access controls. Examples include:

• Do you have a standardized process for building software releases?
• Do you have a formal process to define service levels?
• Do you use a knowledge database of known errors and problems to resolve incidents?

Surprisingly, staffing levels, auditing disruptions, industry and size are not key differentiators. Top, medium and low performers in this study all had roughly the same staffing levels dedicated to IT audit and controls, and Sarbanes-Oxley efforts. And all experienced roughly the same level of disruption due to audit activities. Top performers are also found in the same IT-intensive industries as medium and low performers. Finally, top performers are comparable in size to medium performers, although slightly larger than low performers.

Our overall conclusion, based on analysis of the top performers in this study: IT organizations that frame IT control resource decisions in terms of performance improvement potential and focus ongoing audit- and control-related resources on foundational control activities that have been shown to improve performance measures will generate a significant ROI.

Kurt Milne is the managing director of the IT Process Institute, an independent research organization that supports IT audit, security and operations professionals with independent research, benchmarking and guidance.

ITIL®  is a Registered Trademark of the UK Office of Government Commerce.