THE ROLE OF TPC-D BENCHMARK RESULTS
IN SELECTING A SERVER FOR BUSINESS INTELLIGENCE: PART I
by Daniel Graham
How important are TPC-D benchmark results in selecting a server? The Transaction Processing Performance Council (TPC) offers three important metrics -- Power, Throughput and Price/Performance. A customer shopping for a server on which to run complex business intelligence tasks must keep up to date on this information. A thorough knowledge of recent TPC benchmarks should be a basic component in any prospective purchaser's decision-making process.
However, having said that, a customer must also be aware that many other factors will come into play before he or she can make a final purchase decision. These additional factors emerge at various points in the negotiating process, but the effect is the same in the end: TPC benchmark metrics (TPC-D) become less important as a customer's vendor-selection process matures.
In the first place, a typical purchaser may take as long as nine months to select a vendor -- the average for a mature decision stands around six -- and in that time the leadership in the TPC-D benchmark stakes may have changed several times over.
So, TPC-D benchmark triumphs do not give purchasers a short-cut to an easy answer. In the larger picture of how a mature customer goes about selecting a server, current TPC-D results play a fairly small part in the decision making process. That is particularly true in assessing the likely performance of multi-stream tasks such as data warehousing or data mining.
We'll come to that. But you wouldn't think TPC-D results were one factor among many if you were a fly on the wall observing the early stages of a purchaser's in-house debates or the early stages of across-the-table negotiation. At that stage, the customer regards TPC-D metrics in an all-important light. They are, after all, pure measurements, figures that a customer can grasp and use as a bargaining tool. We know that customers come to the table feeling better equipped simply by having this information.
Let's face it, performance comes high on a typical purchaser's list of priorities. But, as negotiations move ahead, those benchmarks slip back in a purchaser's awareness. They become several factors out of many, and what I want to focus on here is what happens along the way to pull these benchmarks back into the pack.
PERFORMANCE, A CUSTOMER'S NUMBER ONE PRIORITY
When I sit down with a prospective customer, the first thing we do is assign priorities. Performance is their Number One; it scores somewhere around 25% in the totality of their decision-making. This is where the TPC-D numbers fit in, into "performance," and we typically give it a fifteen to twenty percent stake in that particular slice. That is not my decision; it's a decision reached after full and exhaustive consultation with the customer. Now the picture begins to clear. Instead of dominating the negotiations, TPC-D ends up representing about 5% of the total purchase decision picture.
Let's stay with performance a moment more, because there are other factors, such things as concurrency management -- meaning how many users can run queries simultaneously on the system a customer is proposing to buy?
Frankly, TPC-D doesn't help him resolve that question at all. Now let's move on, because a whole lot of additional factors come into play that don't have a straight line relationship to TPC-D results. What range of software is available for a given server? Does the customer have ready access to the appropriate skill-sets among local contractors, ISVs, universities, consultants? Even here the answer is not simple: it depends to a large extent on what database the customer proposes to run on what box.
In my experience, and I have worked to close agreements with a number of
mature customers, the first thing they want to talk about is price. But
after we go at price for a while other factors shake out, and invariably
the top three are SKILLS, PERFORMANCE and SOFTWARE AVAILABILITY. Not
necessarily in that order. The rating changes from customer to customer.
But those are the big three.
---
Part II of this commentary will appear in the next edition of D S * .
---
For more information, see http://www.ibm.com/bi