Analysis & Commentary:

BI INTEGRATION: EXTENDING THE "INFORMATION NET"
by Jim Kanzler, president & CEO, Meta5 Inc

Perhaps the best way to understand the new concept of Business Intelligence Integration (BII) is to equate it to fishing.

Imagine a commercial fishing boat out on the sea. The crew casts a large net into the water and pulls up an enormous load of fish -- all kinds, all sizes. Of course, despite the size of the bounty, there are some fish that escape the clutches of the net. And even though you've got a tasty catch, some of the most succulent fish may be among the ones that got away.

How do you get those fish that got away? Now, think of the net as your company's data warehouse. The warehouse has a wealthy catch of information related to your business. But there are also other bits and pieces of information -- ad hoc data sources -- that do not reside in the warehouse, yet these "data nuggets" are so critical to the company's operations. This data and knowledge includes information contained in ASCII files, Excel spreadsheets, Access databases, legacy systems that didn't quite make the data warehouse list, departmental SQL servers and dozens of other disparate data sources.

There are four "classes" of ad hoc data sources:

• Information that is too small or "out of place" for the data warehouse - it is a minute amount of data that gets used for a specific application and is then thrown away, typically an Excel spreadsheet.
• Information that will be part of the data warehouse, but the planning for its inclusion has not been completed, or the financial resources are not immediately available.
• Information that resides in a legacy system - and is serving a useful purpose - but will never be part of the data warehouse, because the cost or effort to put the information into the warehouse is simply not justifiable based on its perceived value.
• External ad hoc data sources - The tremendous amount of data that we are exchanging with our customers and vendors are representative of the data explosion that we have heard so much about. These ad hoc data sources rarely get inserted into the data warehouse.

Certainly, there are business intelligence tools that -- with varying degrees of difficulty, time, expense, and resources -- are capable of accessing these ad hoc data sources and potentially integrating them into the information residing in the data warehouse. The question, however, is how quickly, how effectively, and at what cost these tools can perform this function. The answer is, frankly, not very quickly, not very effectively, and often at significant cost. This problem becomes magnified when you consider the rate at which these ad hoc data sources are appearing, an outgrowth of the data explosion. As data warehouses were created, companies integrated their pertinent data into these warehouses. However, the number of data sources began to increase, and it became impractical, as well as expensive, to integrate all of the data from these ad hoc sources into the data warehouse.

Enter BII. The theory behind BII was to recognize that there is a problem and address it with a new class of tools. Put another way, BII is a method of focusing the technology of business intelligence to incorporate ad hoc data source as seamlessly and effectively as possible. In speaking at various trade shows, and meeting with several Fortune 500 corporations over the last few years, it has become apparent that the inability of traditional BI tools to deal with these ad hoc data sources was a major problem. It became clear that this key element of the total integration process -- the finishing touch, if you will -- was missing; thus, the emergence of BII.

Industry experts also understand the need for BII, as evidenced by a statement from IT analyst firm, META Group, in a Business Week story dated June 19, 2000:

"The currency of the digital marketplace is intelligence, but it's hard to leverage the cumulative knowledge collected by disparate systems ... A real-time, panoramic, single Customer view (of data) is the Holy Grail ... everyone is searching for it."

Consider this scenario about the tangible benefits of BII technology. Your company's vice president of marketing comes to you on Friday afternoon with a spreadsheet of 5,000 names that his advertising agency has provided, perhaps from a focus group, that relates to customer spending practices and trends. He has jotted some additional information of his own alongside the names. He tells you that he would like to target these customers and prospects through a highly focused marketing effort. You are asked to integrate this information with sales and marketing data residing in the company's data warehouse and to produce a report that will address the viability of marketing to these prospects. And he wants it that same afternoon. You might be able to produce this report through a complicated Visual Basic program or by throwing a team of engineers at the problem, but how cost effective is that? Ultimately, the question is, "With your existing BI tools, can you produce a meaningful report that your vice president really needs within hours, using this ad hoc data?" Probably not.

But using BII, you would be able to create the requested report integrating the ad hoc data source with the data in your warehouse, a process that ordinarily would have taken days or weeks. What's more, it will address the business problem that you sought to answer, in a timely manner that will positively impact your bottom line.

The ability of BII to incorporate data from ad hoc data sources is no more evident than in the case of mergers and acquisitions. A company that acquires another organization may want to analyze a particular product line from the acquired company, integrating their legacy data with your data warehouse to determine marketability and potential market share. Obviously, the information would reside in a separate data source, yet it must be integrated with the acquiring company's market information to produce a meaningful report. BII can accomplish this task almost effortlessly.

Up until now, the focus has always been on producing the quickest query response time. While this is important, what we must also focus on is how to produce the "quickest time to answer" the ad-hoc data integration problem. That is, can you take the information from disparate ad hoc data sources, query it, cleanse it, manipulate it, and produce a report that addresses the specific urgent business questions being asked of it today. Let me ask you, what is your quickest time to answer (QTA) or BII strategy at your business? How much is it costing your organization for not having the ability to integrate all of your data?

That's not to say that building a data warehouse is a wasted exercise: quite the contrary. A data warehouse is necessary for corporate survival in today's competitive environment and represents the best way to assemble most of your data. The problem is, it's just not practical or cost-effective to incorporate every single piece of data into the data warehouse. The key is to have both a data warehouse and a solid BII strategy to deliver all of the information in your company to the people who need it the most.

The truth is every company has ad hoc data sources. The question is how do you integrate these sources with the information that is in your warehouse? Many of us are not even admitting that these ad hoc sources exist; subsequently, it's our job to provide access to these data nuggets in making strategic business decisions.

Perhaps the information from disparate sources will eventually present itself to the point where it is practical to enter it into the data warehouse. The problem is the cost and effort required to enter every last piece of information into the warehouse begins to show diminishing returns. It's clear that companies will take the time and should enter the majority of their data into their warehouse. It's another matter to expect that every data source will be incorporated in a timely manner.

Where is BII going in the future? While the formal concept is still relatively new, the future direction of BII is towards automation and pro-activity.

The ideal scenario is that the information a person wants is delivered automatically, before the person even asks for it. What's more, it can be delivered through the device of choice, such as a pager, e-mail, cell phone, or customized Web portal. The idea is that the system will pro-actively investigate business anomalies, flag them, and report them in much the same way a person is automatically notified when their stock price increases and they are paged with that information. This kind of scenario is already happening, for example, when a person receives an e-mail saying that their airline ticket price has fallen within their desired price range. Why can't this be done with your business data as well? It can be, because BII technology automates many of the engineering processes that were previously done manually.

It's clear that the concept of BII, when applied correctly, can provide answers to business problems that can have an extremely positive immediate impact on a company's bottom line. Many Fortune 500 companies in the consumer package, utilities, computer, and education industries have already embraced BII, and as a result are enjoying its strong business benefits. But how long will it take before other companies come to realize a need for BII, and when they do, will they have fallen behind?

About the Author

Jim Kanzler, president & CEO of Meta5 Inc, has over 15 years experience in delivering data warehouse and business intelligence solutions to Fortune 500 companies.

His company develops and markets business intelligence integration software to solve analytical problems for businesses with ever-changing data and data sources. For more information visit www.meta5.com.