JUSTIFYING A DATA WAREHOUSE PROJECT: PART II
by D. J. Power
Justifying a data warehouse project can be very difficult. We've looked briefly at quantitative and qualitative tools for evaluating a data warehouse project. Also, the upside benefits discussed in part 1 can provide a starting point for a quantitative or qualitative analysis. To explore all of the issues, we need to examine the major analysis of the Return on Investment for data warehouse projects that is being discussed by vendors and consultants. There are some downside issues we need to tackle, too.
In late 1995 and early 1996, Stephen Graham, vice president of software research at International Data Corporation (IDC) Canada, evaluated 62 organizations that had had a data warehouse in operation for at least 6 months. The white paper from the study is titled "The Foundations of Wisdom: A Study of the Financial Impact of Data Warehousing". The title phrase "foundations of wisdom" causes me some concerns, but let's review some of the major finding.
The IDC study showed an average 401% three-year Return on Investment for 52 companies that were not deemed outliers. E. B. Baatz notes "The astonishing numbers are well-documented and compelling enough to make any CEO embark into the data warehousing venue without executing an exhaustive, possibly time-consuming ROI analysis first." In "Case-in-Point" at acxiom.com, the IDC results are also reviewed and summarized. The data warehouse projects had a "mean payback period of 2.31 years. Over 90 percent reported a three-year return on investment exceeding 40 percent. Half reported returns of greater than 160 percent, and one quarter over 600 percent. European companies averaged 340 percent ROI, compared with 440 percent among North American firms.
At the SAS web site the white paper titled "Data Warehousing ROI Tops 400%" (check http://www.sas.com/new/dwsurvey/intro.html ) summarizes the IDC findings. The SAS paper notes "Its goal was to assess the financial impact of data warehousing; its results are nothing short of astounding."
In "Digesting the ROI Paradox," E. B. Baatz argues "the decision to invest in data warehouses seems a lot like the ATM example, with CIOs scrambling to define the ROI and CEOs leaping without the data... The ultimate ROI, if anyone were so enslaved by numbers enough to take the time and trouble to measure it, is astronomically high."
Downside potential
I have read the parts of the IDC study that are available free from a number of the study sponsors. Also, I read the white paper summary. The report can be ordered from IDC Canada by calling Jacquie McLean at 416-369-0033. The methodology used in the study raises concerns for academic researchers. First, the companies Stephen Graham and IDC studied were self-selected by the vendors and IT professional services firms who sponsored the study. Second, the participating firms had to be willing to be identified. Both of these research design elements bias the study so that it is much more likely to find results in favor of building a data warehouse.
Despite this bias, the study seems thoughtful and the authors report some downside potential of data warehouse projects. Also, it should be remembered that the authors examined ROI for completed projects. A retrospective calculation of ROI or payback is much easier and more reliable than an a priori estimate that might be included in a data warehouse feasibility analysis. Let's examine the downside.
Theresa Rigney cautions "be wary of studies like the IDC report. Sure, the 62 completed data warehousing projects examined by IDC reported a great ROI, but most projects out there aren't yet completed. In fact, the costs and complexity of an enterprisewide data warehousing project have capsized a lot of pilot efforts. IT must be aware of these costs -- in time and effort as well as hardware and software -- before it undertakes a full-scale data warehousing project." She argues there "are no ways of predicting or quantifying ROI. The real payback comes from getting information assets out to the users in the best way possible."
In "Case-in-Point" at acxiom.com, the IDC results show eight companies reported negative ROI results, which they attributed to either extraordinarily high costs, low warehouse usage, or a large undertaking requiring longer than three years for payback." Also, IDC "found that discrete implementations -- those developed to support one business need -- realized higher average ROI (532 percent in 1.57 years vs. 321 percent in 2.73 years) than broader, enterprise initiatives. However, the authors were quick to point out that discrete, demand-driven warehouses are in some respects easier to develop."
On a related downside note, Flanagan and Safdie state "An organization's return is measured in productivity and market gains which accrue from the use of the warehouse. Therefore, organizations whose warehouses are unused have the same denominator (costs) as those whose products are well used, but have a much smaller, or nonexistent numerator (benefits)."
On the human upside, Winkler notes that a "data warehouse enables analysts to do for themselves what was previously done by intermediaries." But she goes on to state "The downside of decision-empowerment falls on the shoulders of middle-managers. Their role of data filter is obsolete (check http://www.sentrytech.com/smrwps21.htm )". This possible downside issue is too complex for a brief analysis. Ask yourself if eliminating middle-managers as a data filter and making that role obsolete is a plus or minus.
Finally, some data warehouse projects are failures. According to Flanagan and Safdie, Earl Hadden and Associates found that "85% of current data warehouse projects fail to meet their intended objectives, and that 40% don't even get off the ground". John Ladley, Meta Group Program Director for Application Delivery Strategies, says "We are past the stage where every data warehouse is a success story... believe 40% of the warehousing projects aren't meeting expectations, and 10% are abject failures."
Conclusions
My first conclusion is that a detailed qualitative analysis of a proposed data warehouse is the most that managers can reasonably expect. Although in some situations, financial analysis tools can be useful decision aids, in a data warehouse project their use provides only the appearance of accuracy and precision. When making a data warehousing project decision, managers should generally ask "What are the expected results and benefits?" rather than "What is the anticipated return on investment (ROI)?". Managers should not demand a projected positive ROI from a proposed data warehouse project, but they must demand and anticipate positive results.
When I began writing this commentary I considered arguing that today, in Fall 1997, a data warehouse is a basic business requirement. And therefore executives in most companies, large and small, have no choice but to fund data warehouse projects. I think it is true that data warehouses are a basic business requirement, but I'm not willing to let the IS/IT staff "off the hook" that easily. I'm concerned that some IS/IT managers who want to build data warehouses have insuffient knowledge about them. Please be wary of IS/IT managers who have difficulty explaining in detail the benefits of a data warehouse project. So my more conservative conclusion is that investigating a data warehouse project is a basic business requirement.
A word of warning... Learning enough to understand and explain the benefits of a data warehouse may be expensive. IS/IT managers will need to do more than read a book. IS/IT staff should actually work with a prototype data warehouse. The IT unit may want to hire a consultant; staff should attend seminars and talk to vendors. The process of learning about data warehousing will be time consuming and costly. Companies may need to spend a few hundred thousand dollars on a prototype or a departmental data mart and that is a significant investment. In firms with multimillion dollar IS/IT budgets a prototype data warehouse or data mart project is needed and it should be viewed as "a learning experience". General managers need to spend enough money on a data warehouse project so that IS/IT and business managers can learn about warehouses and can evaluate the benefits. Building a data mart or data warehouse is an employee and corporate development experience.
Your benefits from a data warehouse project will vary depending upon where your company is on the IS/IT learning curve. If your company still relies on a legacy system developed in Cobol, the benefits could be very large, but the implementation risk is high. So start small; focus on anticipated results NOT ROI; but do start a data warehouse project.
References
Last updated November 14, 1997 by D. J. Power. Copyright (c) 1997, D. J. Power.