BRINGING DATA MINING TO THE MAINSTREAM -- PART I
by Inderpal Bhandari, executive editor at large
I am often asked by our customers and employees to explain our mission at Virtual Gold, Inc. My answer is straightforward. Our mission is to bring data mining to the mainstream of application development.
I believe that mission has implications beyond our little company. It is going to be critical for the technology to take hold in general. Allow me to explain. I have now worked in the area of data mining since 1991. At that time, it was described as an emerging technology. Today, it is still described as an emerging technology. When I talk with senior executives of the Fortune 500 companies, many of them are still talking about "evaluating data mining."
I believe that data mining's long drawn-out emergence has to do with one thing alone. Data mining applications cannot be produced by the vast majority of software application developers. As things currently stand, only a narrow segment of application developers that specialize in writing analytic programs, e.g., SAS jockeys, can produce such applications. In this regard, things have not changed that much. A small fraction of application developers have specialized in producing data analytic applications for at least thirty years. These developers have special training in writing data-analytic programs, and are, deservedly so, a prized commodity amongst employers.
Contrast this situation with that of creating database applications. A graduate fresh out of school can produce a sophisticated database application in no time. The knowledge of how to do this is widely available and there are tools that come with the popular databases that allow for quick turnaround on application development. All in all, this has allowed database applications to become ubiquitous.
However, data mining platforms are not amenable to use by the vast majority of software application developers for a number of reasons. For example, the mathematical complexity of the underlying algorithms rule out most developers who have not undergone advanced training in statistics or other data-analytic disciplines. Be that as it may, the bottom line is that as things stand, the technology is not going to enter the mainstream of application development.
Our mission at Virtual Gold is to change this situation fundamentally. We want data mining technology to become as pervasive as database technology is today. We want the technology to be in the mainstream of application development and not just confined to a segment that specializes in business analytics. I believe that is the way to realize the fullest potential of this technology.
I recognize that this will be a very tough nut to crack, and I will have more to say on exactly how we propose to do this in future columns. Presently, I will close by describing the traditional approach to attaining our objective of bringing data mining technology to the mainstream.
There are other data mining companies that share our common goal of getting data mining to abandon the shrinking violet act and to finally, emerge. But they have adopted the more traditional approach to attain that goal, namely, to build vertical applications that target specific domains. Having helped pioneer this approach by creating one of the earliest vertical applications of data mining in the IBM Advanced Scout program for the coaches of the National Basketball Association, I have been in the belly of this beast. Next week, I will argue that its considerable strengths not withstanding, it will not make data mining a mainstream technology.
---
Inderpal Bhandari can be reached via http://www.virtualgold.com