DATA MINING: IT'S NOT CHESS
by Zak Pines

When Bruce Springstein sang of the human touch, I'm sure he wasn't referring to business intelligence.

But, as I will show you, the Boss's words ring true when it comes to data mining.

Mining a business's data -- whether that business is a basketball team or an insurance company -- is very different from using a data crunching system to win a chess match.

Data mining isn't chess. In chess, a computer on its own was able to outperform the top chess player in the world, Gary Kasparov.

In data mining, a computer can only do part of the job. This is the reason that Advanced Scout, IBM's data mining program for National Basketball Association teams, has had such success in its few years of existence. The program was structured in such a way so that a non-technical domain expert -- in this case an NBA coach -- could use the program to find patterns in his data, and then use his basketball knowledge and intuition to interpret and act on the patterns.

This should be the business plan that companies follow when incorporating data mining into their strategy.

A GOLDEN OPPORTUNITY

The need for a domain expert coupled with a data mining program should not be viewed as a hindrance but instead as a golden opportunity. Data mining is just one more tool, albeit a very valuable one, in the decision making process. This is why an interface whereby a non-technical lay-user can interact with such a program is so vital to its success.

If you were an NBA coach, would you hand over all of your team's decision making to a computer, even if it was the most powerful, efficient and artificially intelligent one in the galaxy?

Of course not.

What data mining can do is help the coach -- or the executive or the manager -- evaluate his past decisions and plan his future ones. But a data mining application cannot make these decisions on its own.

The most basic reason for this from a business intelligence standpoint is simple -- there's no reason to limit yourself to the data you've collected.

If your data collection is not perfect, you will need human intervention at some level. And, most likely, your data collection will not be perfect. In chess you can capture all of the variables, because there are a clear and finite number of them, but this cannot be done in real life.

A BRIEF WALK

To see why, let's take a brief walk through the data mining process.

First, a company collects data using several attributes. Then, upon setting up different analyses, the user can run data mining to find hidden patterns in the data. Now comes the need for the human touch.

The data mining pattern will tell you something about what has happened in the past. The pattern itself is neither a prediction nor a promise. On its own is says nothing about what will happen in the future.

Instead, the domain expert can use this and other previously hidden patterns as he anticipates what will happen in the future.

For example, if the data mining application discovers that an aspect of the data has been surprisingly successful, such as a lineup in basketball, the user then has to make a decision. Do I make this surprisingly successful thing more prominent? Should I play this lineup more?

Not necessarily. If the pattern becomes more prominent -- if the lineup plays more minutes -- it may perform at a less successful level.

THE TINKER

A second option is to tinker with another aspect of the data -- another lineup in the basketball example -- to make it more similar to the successful thing. For example, in basketball, maybe one lineup is running a certain play one way and another lineup is running it differently.

Or maybe a coach -- with the knowledge that lineup one is outperforming lineup two -- wants to tinker with lineup two by making a substitution in it to make it even more different from lineup one, in an attempt to counteract its current weakness. Data mining can help trigger these thoughts in a coach's mind by giving him a more in depth understanding of what has happened in the past.

As you can see, the difficulty for the coach or the manager or the executive comes from deciding what and how to tinker. First of all, you may not have collected this information in your data. So the computer will not have a say as to what to do next.

But the computer has already done its job, in that it has discovered a pattern in your data. What you can do now is go back and look at the situation with this fresh perspective. This new perspective may allow you to make a more informative and successful decision. That is the value of data mining.

Data mining can tell you what happened, but it isn't a crystal ball. Data mining is only as good as the domain expert using it. It is neither perfect nor a guarantee. But it's going to help. And the lesson to keep in mind is that it's going to help even more when used with the human touch.

---

For more information, see http://www.virtualgold.com