LEVERAGED SOFTWARE SELECTION: A BETTER WAY TO SELECT DATA WAREHOUSE TECHNOLOGIES
by Benjamin Taub, Dataspace, Inc.

Last month I attended a data warehousing strategy session for the IS branch of a large airline. The manager of each of the company's data warehouse projects was asked to give a brief presentation on the status and findings of his or her project. We can leave the semantic question of whether these projects were actually creating data warehouses or data marts for a later time.

Three of the projects were building databases to be accessed with relational OLAP tools. One topic covered by each of these three was the process used to select a relational OLAP (ROLAP) tool. Most interesting to me was:

• Each of these groups went through a lengthy, costly evaluation and benchmarking procedure
• Each selected the same tool, and
• Each ended this section of their presentation by stating that, "by the time our evaluation was completed, the market had shifted and we were no longer sure that our choice was still the best."

There were no special, industry-specific needs here. These projects were each evaluating standard, off-the-shelf technologies. While this was somewhat of an extreme case, similar episodes are repeated daily. Companies fail to leverage not only the work done by the industry as a whole but, sometimes, even that done by other projects within their organization.

We are constantly driven by a search for the 'best' answer. But, what exactly is best? One tool evaluation that seems to be performed by virtually every IT organization compares Cognos' report writer, Impromptu, with that of Business Objects Corporation, Business Objects. I contend that, for the vast majority of situations, either tool would suffice. And, as each company leapfrogs over the other, today's correct choice becomes tomorrow's mistake. What is really gained by devoting months to comparing these tools?

How does the evaluation process usually work? Companies set up lists of tasks and run each candidate tool through those tasks. Because the evaluators usually have little experience with these tools and limits to the time they can spend with each, these evaluations are generally not very deep.

Thus, important points are inadvertently overlooked, important points that become painfully obvious once the tool is purchased and used. For example, one of the market leading data movement engines allows only one developer to access a subject area at a time. This is clearly a problem when multiple developers are working on one project. As you might expect (unless you are incredibly naive) the product's sales force does not alert potential customers to this fact. But, how many of these potential customers include 'accessible by multiple simultaneous developers' in their test suite? Probably none; such capability is usually assumed. Thus, evaluations fail to turn up vital information. The only valid test is day to day application of the technology.

Therefore, the standard approach to evaluating warehouse technologies is wasteful and ineffective. But, without some sort of evaluation process you could be accused of not showing "due diligence." What to do?

Most of the tool categories (ie. data movement, OLAP, query, etc.) in the data warehouse arena have clear market leaders. They became leaders because other companies did extensive evaluations, purchased and then either purchased more or dropped the tools based on actual experience. Why not utilize this knowledge as it is reflected by market share? Why not leverage the efforts of these other companies?

The leveraged software selection process can lead to better technology choices in much quicker timeframes, usually two to three weeks. The leveraged approach generally proceeds as follows:

1. SOLICIT EXPERT INSIGHT: The desire to develop an internal skill base should not preclude you from soliciting the help of experts who have, "been there before." Remember, you are trying to develop that internal skill base, you may not already have it.
2. CREATE A LIST OF KEY SELECTION CRITERIA: With the aid of your expert and any staff that will be affected by the tool, develop a list of requirements which the tool must satisfy. Developing these lists is an art, not a science. We prefer limiting these lists to 5 pages. Any more is difficult to manage.
3. SHORT LIST CANDIDATE TECHNOLOGIES: Develop a list of the two to three market leading products. Extend this list to include products recommended by your expert or that seem to have unique, market-leading capabilities.
4. REVIEW HANDS-ON EVALUATIONS: If available, obtain comparative test results that include the tools on your short list (for example, we offer a hands-on comparison of the leading engine-based data movement tools and will be offering other comparisons in the future). It is also useful to call reference sites at some time during this process. While the input of traditional IT strategy firms is useful, be careful, the opinions of these companies are often based on vendor literature and meetings. These firms frequently have not actually used the tools.
5. HOLD VENDOR DEMONSTRATIONS: Ask each vendor to discuss their strengths and strategy. View a demonstration of each tool to gain a better feeling for its capabilities. Press the vendor to answer any concerns that your investigations uncovered. Finally, review your list of criteria with the vendor.
6. SELECT A TOOL: Given the information obtained through this process, select a primary candidate tool. Remember, there frequently is no 'best' tool. Each may shine in different areas.
7. PROOF OF CONCEPT (OPTIONAL): If desired, perform a one to two week proof of concept with your chosen tool. Create a test plan that models your situation. In fact, you may want to attack a portion of the system you are attempting to create. Request that the vendor supply a resource to help you with this test.
8. NEGOTIATE A PURCHASE: Correct implementation of this step is particularly important in the leveraged software selection approach. Negotiate a purchase that provides liberal return terms (1 - 2 months). Leave yourself enough time to thoroughly implement the tool before your return period expires. Thus, while many companies will spend 1 - 2 months learning and doing ineffective tests of candidate technologies, your final evaluation is an actual implementation of the chosen tool. If it works, as it likely will, then you've accomplished 1 - 2 months worth of work. If it doesn't, you've lost little or no time in comparison with the traditional selection approach but have gained strong experience in that tool category.

The leveraged software selection approach will work best for tools that have large potential customer bases. These customers will reward the best players with large market shares. This approach is less useful for specialty tools that do not share such broad bases of potential users.

The leveraged software selection approach is a tool that could be profitably added to the toolbox of most IT organizations. It replaces the time & money wasted in useless evaluations with time put into completing projects.