GOLDEN MEANS: DESPERADOES AND MANAGERS
by Inderpal Bhandari, executive editor at large

For the last eight years or so, I have seen data mining described variously as an emerging area or a leading-edge activity. A recent report based on a survey conducted by Price Waterhouse and the Conference Board talks about the slow uptake of data mining tools. Of particular interest was a comment that it seemed too much to ask organizations to address data mining in the initial phases of a data warehousing project. I guess the labels emerging and leading-edge are here to stay.

The slow adoption of data mining has mainly to do with the demands the current state of the technology places on its users. In this article, I will illustrate some of the challenges faced in staffing and managing a data mining project.

Let's start by refreshing your memory. Last week, I described how the use of data mining led the coaching staff of the Orlando Magic, a National Basketball Association team, to think out of the box. Playing with a team depleted badly by injuries, they found via the IBM Advanced Scout data mining program that when they played a third-string guard along with a second-string center they had outscored a tough opponent, the Miami Heat, by 23 points! They centered their game plan around the guard, Darryl Armstrong, a change that allowed them to beat the far superior Heat twice in last year's playoffs.

Note that while the data mining program found the interesting pattern automatically, it was the coaching staff that interpreted the pattern, i.e., determined that it was a useful pattern and translated it to something actionable, namely, a game plan. This is not an easy task. It requires detailed knowledge of the underlying domain, namely, basketball.

Although not so in the case of Advanced Scout, such interpretation of the output of many data mining programs can require one to also have knowledge of statistics and data analysis over and above knowledge of the subject area. Ideally, a data mining project manager should describe his team with a quote from the Bible, Book of Psalms: Behold, how good and how pleasant it is for brethren to dwell together in unity. Under the above circumstance, where both data analysis expertise and business expertise must be present in the team, this may not be possible.

Inherent in the composition of the team lurks a cultural difference that may work against such unity. On the one hand, there are analysts. They understand the data mining software and the process of data analysis. They are usually young , sharp, and fresh -- from graduate school, that is. On the other hand, there are business specialists, who understand the business. They are usually no-nonsense veterans, seasoned and proud of their hard-knock schooling.

And, now to add fire to this combustible mix. Both camps have to contribute to the interpretation of results of data mining, the analysts from a statistical perspective and the business specialists from a physical perspective. In the context of knowledge discovery, it is a known fact that what is statistically significant may not be physically important and vice versa. Which implies that these two camps will often reach different interpretations and must be prepared to resolve differences. I don't think I need to complete the picture of the chaos that can ensue.

So, what is the project manager to do? It's easy to appreciate that he is in a tough spot. Roy Blount, Jr. once explained why he was so embarrassed to say he was a writer. 'If you were a member of Jesse James' band and people asked what you were, you wouldn't say, "Well, I'm a desperado". You'd say something like "I work in banks" or "I've done some railroad work".' Should the manager favor one camp over the other, it could crush the morale of the team, leading to the creation of writers, and perhaps, even desperadoes. What if he splits the difference every so often, siding with one group and then the other? That strategy is unlikely to work, either, since in most situations, one camp will definitely be right. Or, he could approach the issue in a different manner, saying, "let the analysts crunch through the numbers first, and then the business guys could take a look at the results". This approach also may not work. The analysts, not being fully aware of business realities, could filter out results that are not statistically significant but which may still be physically important.

The bottom line is that staffing and managing a data mining project is challenging for reasons that are intrinsic to the current state of data mining technology. I will have more to say in future articles on how the technology must evolve to meet that challenge, it being a major focus of our work at Virtual Gold. That is not to suggest that tremendous benefits are not possible with the technology as it currently exists. In fact, to the contrary, early adopters can reap tremendous savings and competitive advantages, as has been illustrated by several case studies. It is simply to suggest that in each of those cases, the technology was not the only story. There was also a talented team in place that overcame the challenges and was key to the success of the project.

Which brings me full circle to the Orlando Magic. Currently, Orlando is struggling to qualify for the play-offs. Their injury situation is worse than it was last year, and to make matters worse, their secret weapon, Mr. Armstrong, is among the injured, out for the season. Can data mining help Orlando find another combination that will save them?

Yes, it can but not if they run out of players. There is just so far that technology goes. Eventually, it's the quality of the players that you have on the floor that wins the game.

Interested in learning more about staffing and management of data mining projects?
Contact us: ( http://www.virtualgold.com ).