BRINGING DATA MINING TO THE MAINSTREAM -- PART II
by Inderpal Bhandari, executive editor at large
Last week, I mentioned two approaches to making data mining a mainstream technology. One is the traditional approach of building vertical applications that target specific domains. The second is the approach that we are following at Virtual Gold, namely, to make data mining technology amenable to use by mainstream software application developers. Having helped pioneer the traditional approach by creating one of the earliest vertical applications of data mining (the IBM Advanced Scout program for the coaches of the National Basketball Association), it is clear that my viewpoint has evolved. Its considerable strengths not withstanding, I no longer believe that the traditional approach alone will make data mining a mainstream technology.
As was clearly demonstrated by Advanced Scout, the main advantage of integrating a data mining component within a vertical application is that the user interface can be simplified to the extent that an end user can mine data. The user is no longer directly exposed to the mathematics of the underlying algorithms but instead deals only with business concepts that he or she is already familiar with. Put another way, such integration of data mining technology allows it to effectively reach non-technical users, thereby broadening the set of users of data mining technology. It also leads to a good business model in the short run since there is a clear focus on who the customer is.
Clearly, if vertical applications with an integrated data mining component were developed for a large number of domains, data mining would indeed become a mainstream technology. This is the direction that a small but significant group of vendors is hoping will take hold in the finance, and more recently, the customer relationship management areas.
The problem that I have with the above approach is not that it will not succeed eventually, but that progress will be made at a glacial pace. I know first hand that integrating data mining with a vertical application entails a lot of knowledge engineering which is very labor intensive. It is no accident that there are whole companies that specialize in producing only one such vertical application.
This has two implications. First, it takes a long time to create such vertical applications. Second, the high cost of labor-intensive production is eventually passed on to the customer. A hefty price tag translates to considerable evaluation and justification, even on the part of a senior executive. Taken together, these two phenomena suggest that the traditional approach will entail -- indeed, one could say has entailed -- a very long adoption cycle for data mining technology. And, as we all well know, in the computer industry, a long adoption cycle often means no adoption at all. For these reasons, I believe that the traditional approach alone will not succeed in bringing data mining into the mainstream.
We also need a way to engage the mainstream software developers. If mainstream developers get involved in creating data mining applications, the technology will take hold in a flash. By way of example, consider the situation in a large corporation. There are a large army of programmers who develop in-house applications, e.g., payroll, surveys, sales reports, etc. Imagine what would happen if all these developers suddenly had the means and ability to rapidly integrate data mining with their applications. Data mining would take hold in that organization instantaneously, leading to all kinds of near-term and long-term benefits.
However, current frameworks for developing data mining applications are suitable for use only by the developer skilled in analytic disciplines, e.g. the SAS jockeys. They are not amenable to use by the average developer. Consequently, there is a clear need for an application development framework that can be used by the typical software developer to produce sophisticated data mining applications. The VirtualMiner(TM) framework to be announced by Virtual Gold in the near future is designed to fill this need in the marketplace. Will it succeed commercially? Only time will tell. Be that as it may, our little caper will be valuable, much the same way a scientific experiment is. We will soon know if this is a viable approach to speeding up the adoption of data mining technology.
---
Inderpal Bhandari can be reached via http://www.virtualgold.com