THE NEW FACE OF BUSINESS INTELLIGENCE
by Dan Kara

E-commerce, ERP, hosting, and portals are the latest frontier for business intelligence tools.

During the second half of the 20th century, the United States and Canada, Western Europe, as well as Japan and a number of other countries, have evolved from primarily manufacturing-based economies to more information-based societies. Business managers in these countries understand this trend and recognize its logical consequence; namely, that information is a key corporate asset, following only people in terms of importance to the organization.

A variety of products, technologies, and techniques are available to business and technical managers that make it easy to get data into the hands of managers and other decision makers, and to foster information exchange among workers. One class of products, and for lack of a better term we will call them business intelligence (BI) tools, transforms data into information and presents that information to end users in a meaningful and usable manner. Most products in this class also support advanced analytical techniques and free-form exploration of data, thereby converting raw data into information, and information into knowledge.

This class of products has undergone a fundamental change recently. BI products and technologies are now being employed in ways that transcend their historical role, particularly as they apply to the New Economy. Business intelligence tools are now central technology for e-commerce customer analytics, information portals, ERP enhancement, and myriad other roles. They are also finding favor with a new class of end users. As such, the market for BI tools is expected to surge. Market researcher International Data Corp., Framingham, Mass., estimates that the market for business intelligence tools alone will reach $5.2 billion by 2003, to say nothing of the other classes of products (data warehouse hardware and software, for example) and services necessary for them to operate. Dataquest, San Jose, Calif., is even more bullish, estimating that the business intelligence market will reach $6.2 billion by 2002.

To understand how and why business intelligence tools are extending their reach, it is important to understand their historical roots and general selection criteria. The term "business intelligence tools" is somewhat ill defined, but in its favor it does give broad coverage to a vast array of products with different historical backgrounds. Over time and with each IT paradigm shift, new BI products have come online and existing products have been reworked to offer new functionality, along with additional areas of applicability and reach. However, there are core pieces of functionality that differentiate the various classes of products which will, to greater or lesser degrees, determine their effectiveness and ability to be applied in new functional domains.

BI 101

In simpler times, say five or six years ago, BI toolsets could be broken down into two distinct camps: Executive Information Systems (EIS) and Decision Support Systems (DSS).

As the name implied, EIS tools created applications that could be used by nontechnical business managers to drill down and analyze highly summarized data to support decision making. EIS systems typically ran on mainframes, were pricey (over $100,000 for the server component), and largely the exclusive property of top-level managers.

DSS tools, which were relegated to more technical knowledge workers, supported ad hoc data access, advanced analytical techniques, and were relatively cheaper.

Changes in the EIS and DSS tools marketplace, the result of complementary technical and business trends such as the rise of client/server and Internet computing, and the flattening of corporate hierarchies with autonomous business units responsible for their own bottom line, made the earlier distinctions moot.

Line-of-business management and knowledge workers now require ready access to corporate data to increase the effectiveness and competitiveness of their business units, much like their counterparts in upper management needed information for strategic decision making at the corporate level. Also, to remain competitive, EIS tools were retrofitted to run in client/server followed by Internet environments, with an accompanying reduction in cost.

While these changes were taking place, a new generation of client/server query and reporting products were brought to market. Like EIS/DSS toolsets, these products provided database access with limited coding and also created "read-only" applications, but as their name implied they were primarily reporting tools, although they did provide rudimentary analysis capabilities. Not surprisingly, EIS/DSS products and database query and reporting tools evolved separately into a new class of products that offer both analytical and reporting functionality, mimicking a biological phenomenon known as "convergent evolution." Web enablement and ease-of-use enhancements are core to all classes of products.

Multidimensionality Wars

The purpose of business intelligence tools is to allow business users to analyze, manipulate, and report on corporate data using familiar, easy-to-use interfaces. Obviously, information should be presented to the user in a way that conforms to a businessperson's understanding of the data. Unfortunately, in relational database management systems, data is structured in rows and columns that run across multiple tables. Also, the data is highly normalized to prevent redundancy, guarantee integrity, and provide for data access flexibility. While collections of normalized database tables are readily understandable by DBAs, data in this form offers little comfort to the average business user.

All BI vendors agree that users must be presented views of data in a highly summarized, "cross-tab" form that supports comparative analysis. Basically, there are two different approaches to presenting business users with information in a way they can understand. The first involves multidimensional storage of data. The second method requires normalized data structures to be "reconstructed" into a "view" that maps directly back to their original business representation.

Multidimensional databases (MDBMS) store data in matrices along related attributes, or dimensions, that are common to the business, such as product, time, cost, geographical spread, sales, etc. Because there can be many dimensions in a multidimensional data store, they are often referred to as "hypercubes" or simply "cubes." Business intelligence vendors supporting multidimensional databases can be distinguished by the manner in which they handle sparse or dense data (if the data in the hypercube is mostly empty, the cube is described as "sparse"), and how the cubes are stored on disk. Most vendors use complicated indexing, hashing, and compression algorithms to reduce the size and optimize access to the cubes. Other vendors write the cubes directly to disk (as multidimensional arrays). As cubes require N! indexes per N dimensions, this limits the size of cubes implemented this way.

Another approach allowing users to view data in terms familiar to them is to provide multidimensional representations of relational databases by creating multidimensional "logical views" of relational data. Vendors have added to their tools a metadata layer that hides the physical structure of the normalized database from end users and maps business terms to their underlying relational database representation. This is accomplished by combining tables that are commonly linked into a single logical table, and mapping table and field names to more user-friendly equivalents. Vendors also utilize multidimensional indexes and hash tables, as well as optimized SQL, to overcome performance bottlenecks.

The tools can be further subclassed based on their ability to pull information off the working database or data warehouse into smaller, local databases (relational or multidimensional) on the desktop. The tools then feed off the local data sources to create reports, perform "drill downs" and "what if" analysis, as well as perform other simulation and forecasting techniques. Some tools provide no specialized data storage mechanism, but provide multidimensional views of transactional data stores or data warehouses. Others provide their own local relational stores that can hold information "extracted" from transactional data stores or data warehouses.

A third class of tools extracts subsets of relational data and stores them in multidimensional structures. These multidimensional extracts differ from a full MDBMS in that they are typically local to the desktop, limited to smaller amounts of data (often all cells of the hypercube are stored on disk with no indexing or hashing), and lack some advanced MDBMS functionality.

The fourth class of BI tools is centered on high-end, typically server-based, multidimensional databases. The MDBMS is populated from extracts from operational databases and data warehouses.

Other Selection Criteria

In addition to the structure and location of data stores, BI products can also be characterized in general terms according to where computational logic and data manipulation functionality are executed. Vendors offer a variety of two-, three-, and multitier execution architectures. Generally, products that support only two-tiered software architectures are less expensive than more complex, multitiered tools. In general, they also provide better visualization techniques and stronger integration with desktop productivity packages. BI tools capable of creating multitiered applications are better positioned to build high-performance, complex applications, serving large numbers of users.

Other tactical selection criteria among BI tools include the ability of end users and corporate strategists to easily explore data to seek out new market opportunities and drive the business. The most common type of BI navigational functionality allows users to view data from a variety of perspectives. This so-called "slice-and-dice" capability permits users to view data across any dimension or combination of dimensions within a hypercube (or logical "view" of a hypercube) whether they are directly linked or not. Virtually all BI tools also allow users to graphically "drill down" through layers (drill levels) of consolidated data to more detailed layers of information using simple point-and-click. A piece of functionality that is related to both slice-and-dice and drill-down is commonly known as "traffic-lighting," where users can associate visual prompts or execute other actions when some user-defined limit for data values is exceeded.

Some of the latest, more cutting-edge navigational functionality automates the exploratory process replacing, to some degree, the human data analyst with "intelligent agents." Using intelligent agents, exceptions and other unusual data come directly to the user, instead of the user looking for them. In the best tools, rules-based software agents run asynchronously in the background filtering and monitoring data, performing calculations, and comparing results. Exceptions and other interesting data are automatically posted to the user for further analysis.

Analytical Functionality

The four most common types of analytical functionality provided by BI tools are estimation, forecasting, time-series analysis, and modeling. Estimation, forecasting, and time-series analysis are usually based on linear regression techniques. EIS/DSS tools that support modeling generally provide estimation, simulation, and forecasting techniques that operate on nonlinear variables.

Forecasting and time-series analysis differ from simpler estimation techniques in that with the former, future trends are predicted based on existing historical data. Also, unlike simple estimation, using forecasting and time-series analysis the users are often given a choice of a variety of line-fitting techniques. Forecasting and time-series analysis differ little except that with time-series, data regressions residuals are correlated over time, with the result that a slightly different set of regression techniques must be employed.

BI tools present data to end users in a readily understandable form. This implies the use of familiar idioms including spreadsheets and popular report and graph formats to make complex data understandable. BI tools, therefore, should support common business report formats such as cross tab, columnar, forms, summary, and master/detail. Most tools support the common banded report editor, but current state-of-the-art report writers allow editing of reports using "live" data. To prevent the database from being "hit" each time an edit is made, tools typically provide a caching mechanism for storing downloaded data.

No single class of tools is superior. It is all a matter of what you are trying to accomplish, and the price you are willing to bear. This simple rule applies whether or not traditional BI functions and execution environments are targeted, or if the tools are applied in new ways for use in information portals, as a hosted solution, or for e-commerce and packaged application analytics.

Information Portals

The emergence and effectiveness of Internet portals such as MyYahoo, which aggregate and make readily available large amounts of "personalized" information, have resulted in pressure to apply the same idea at the corporate level. Both business and technical managers now view enterprise portals as the best means of delivering valuable personalized business data and content directly to employees, business partners, and customers. As a result, corporations are increasingly deploying portals on corporate intranets to enable users to find information in a categorized and organized fashion. Likewise, companies have realized that B2C relationships can be enhanced by providing information (e.g., product specifications, technical notes, and service information) in a portal-like structure personalized for a particular customer's needs. B2B online trading communities, which are appearing on a daily basis, are almost exclusively based on some type of portal design. In fact, as organizations build new classes of Web-based applications, engineered using a new generation of tools, technologies, and techniques, they typically sport a portal as the front end.

Portals are now the Web user interface to deliver personalized, aggregated information encompassing all aspects of business transactions and processes to employees, customers, and business partners. Portals are also the perfect front ends for business intelligence applications. Vendors of BI products have responded with decision portal framework products, or have added portal front ends to existing BI products, that exploit the power of portals.

Decision portals provide a user-friendly, Web-based single point of entry to a variety of reports, charts, and analytical applications, as well as online analytical processing (OLAP) and report servers. For example, Palo Alto, Calif.-based Brio Technology Inc.'s Brio.Portal exposes a number of different functions, including accessing structured and unstructured data, analytical and drill-down capabilities, real-time reporting capabilities, distributed reporting, as well as real-time messaging, in a customizable Web interface. Other BI vendors that have released portal products and frameworks include Sterling Software (acquired by Computer Associates); Cognos Inc, Burlington, Mass.; Visual Mining Inc, Rockville, Md.; Business Objects Inc, San Jose, Calif.; Hummingbird Ltd, Toronto; and SAS Institute, Cary, N.C.

BI portal products are not limited to classical BI functionality. Most products also provide some type of content management functionality, along with the ability to customize their interface. The later piece of functionality cannot be overstressed. The goal of decision portals is to provide a single, personalized interface to aggregated information. They are another step in the evolution of BI products from their limited role as specialized tools for highly trained analysts, to tools for business management and knowledge workers to make better informed decisions.

The real need for e-commerce analytics has provided existing vendors with new markets, as well as launched the next generation of BI players.

E-commerce Analytics

With the rapid growth of e-commerce initiatives across virtually all industries, business intelligence practices have found new areas of applicability and usage within untouched vertical market segments. Organizations amass mountains of data generated from B2B and B2C interactions, and this data is ripe to be analyzed and acted upon. The application of business intelligence practices to e-commerce falls under the aegis of "e-commerce analytics."

Organizations are acutely aware that in the Internet Age the difference between success and failure is measured using metrics of increasingly smaller increments. E-commerce analytics represents a rare opportunity for an organization with an e-commerce presence to optimize its operations, enhance relationships with customers and trading partners, and gain bottom-line benefits. The real need for e-commerce analytics has provided existing vendors with new markets, as well as launched the next generation of BI players. At this time, however, the market is quite young and a number of solution providers that "own" pieces of necessary functionality are partnering with others to deliver a complete solution. Solutions for analyzing click-stream data illustrate the phenomena.

A click stream is the sequence of clicks or pages requested as a visitor explores a Web site. Click-stream data, which tracks the movements and preferences of Web site visitors, is generated in abundance from e-commerce interactions among customers, trading partners, and suppliers. Analysis of click-stream data yields demographic (age, gender, income, etc.), geographic, and psychographic (preferences, propensity to buy, etc.) information that results in a detailed understanding of customer buying patterns, behavior, and preferences, as well as the same from trading partners, suppliers, and other business partners. Capture and collation of click-stream data is provided by Web usage analysis vendors such as WebTrends Corp, Portland, Ore.; NetGenesis, Cambridge, Mass.; and Accrue Software, Fremont, Calif.

To bolster their analytical capabilities, Web usage analysis vendors have incrementally added more robust analysis functionality to their products, but most have partnered with traditional BI vendors (they have also partnered with e-commerce engine vendors such as Broadvision, Vignette, and others). For example, Accrue and MicroStrategy Inc, Vienna, Va., have partnered to integrate the MicroStrategy OLAP engine into Accrue's Accrue Vista. MicroStrategy also has a partnering relationship with NetGenesis. For its part, WebTrends has an agreement to resell the Hyperion Essbase OLAP server from Hyperion Solutions, Sunnyvale, Calif. Over time BI vendors will add their own click-stream analysis functionality to their product sets, often by way of acquisition.

Web advertising and profiling vendors, which represent a major class of click-stream data generators (along with e-commerce engines and other sources), have also partnered with BI vendors to bolster the analytical capabilities of their offerings. These vendors include DoubleClick Inc, New York City; Be Free Inc, Marlboro, Mass.; and Engage Inc, Andover, Mass. DoubleClick's Databank solution, for example, which allows online advertisers to capture, store, and analyze key information related to their online advertising campaigns, sits on top of the Hyperion Essbase engine.

Hosted Solutions

The Internet and related technologies, such as browsers, HTML, TCP/IP, IIOP, and now XML, have made it possible for a new outsourcing paradigm to emerge where software is rented or leased (as opposed to purchased). The applications themselves are centrally managed by an Application Service Provider (ASP) and can be accessed from anywhere on the planet using standard IP networks, including the Internet, extranets, and virtual private networks (VPNs). In addition to hosting, ASPs are responsible for the smooth deployment and operation of the software, including ongoing maintenance, software upgrades, customer support, and occasionally training.

The application hosting approach allows companies to outsource the deployment, operations, maintenance, and management of their software systems, thereby freeing them to focus on their core business competencies. For cost-conscious business management, hosting offers reduced capital outlays for servers and software, as well as predictable monthly usage fees, with the net result that cash flow is greatly improved. Moreover, companies have the benefits of using leading-edge software without the necessity of installing and maintaining it, to say nothing of the armies of increasingly rare and increasingly expensive trained staff required for the care and feeding of such systems. Given the business and technical advantages of the hosting approach, the market potential is expected to be huge.

Initially, most hosted applications fell into four basic food groups, including e-commerce (B2B, B2C, Web advertising, procurement, billing, etc.), office automation (e-mail, groupware, collaboration/collaborative planning, virtual office solutions, etc.), back-office (payroll, HR, supply chain management, and electronic payment), and vertical market (legal, finance, real estate, etc.) systems. ASPs and vendors of BI products are joining the growing ranks of service and software firms offering hosted solutions, including business intelligence solutions (and implicitly data warehousing services).

Vendors such as Business Objects, Minneapolis-based Net Perceptions, Cognos, and others have partnered with ASPs or have released special development kits or "starter packs" to facilitate the deployment of hosted BI solutions. ASPs will often work with multiple BI solutions, while purveyors of BI products, casting the largest ASP net, typically partner with multiple ASPs. WhiteCross Systems, a U.K.-based ASP, supports solutions from Brio and Angoss Software Corp., Toronto. Conversely, BI vendor Cognos has partnered with ASPs Corio Inc, San Carlos, Calif.; and Infinium, Hyannis, Mass.; among others, while Business Objects has announced relationships with ASPs including NewChannel Inc, Redwood City, Calif.; Corporate Systems Group Inc, Miami; Mitratech Inc, Los Angeles; and RZ Solutions, Minneapolis.

Due to a reduction of upfront capitol expenditures for software, servers, and other hardware, hosted solutions appeal to the small-to-medium enterprise (SME). Hosted BI solutions, especially the more complex, robust, and expensive (those sporting server-based multidimensional databases, for example), are now open to a whole new class of business that in the past had neither the deep pockets nor the technical wherewithal to support them.

Packaged Apps Analytics

The BI market has recently been given a boost with the realization among business managers that the boatload of information generated and stored within enterprise application packages, including but not limited to ERP and CRM solutions, was not as highly leveraged as it should be. Response to this oversight has come from two corners - the packaged application vendors and traditional BI solution providers.

Packaged application vendors were only too happy to respond to the call for analytical support within their tools. First, the need was real, and second, they were searching for ways to overcome declining ERP revenues due to market saturation. Most of the larger enterprise package application vendors, including Oracle, SAP, and PeopleSoft, have released or have announced new analytical modules that can be directly integrated with their existing tool suites. These products tie into the data warehouse solutions provided by the enterprise packaged app vendors and initially provide financial-analysis, balanced-scorecard, and other business performance measurement applications. As a case in point, PeopleSoft now offers its Customer Relationship Management Analytics suite, which comprises the PeopleSoft Customer Profitability, Balanced Scorecard, Sales Effectiveness, Support Effectiveness, and Marketing Effectiveness workbenches. These workbenches utilize the PeopleSoft Enterprise Warehouse, are innately Web-enabled, and provide a number of customizable analysis and reporting tools and templates.

Traditional BI vendors also play in this space. Many vendors provide interfaces into packaged applications and have partnering relationships. While it would seem that the two vendor classes would be fighting for marketshare, the BI vendors often provide advanced functionality not available in the standard packaged applications, including advanced reporting capabilities and statistical support such as forecasting and time-series analysis.

BI products are now in the midst of another evolution and repositioning, adding workforce planning, best-practices analysis, and business and revenue process optimization support, just to name a few.

No End In Sight

Corporate management understands the importance of the data collected day-to-day during business operations. Organizations that can exploit their corporate data to respond quickly to internal and external events are more competitive than those that do not. Business intelligence products, including generic BI solutions, desktop and enterprise query and reporting tools, multidimensional databases and data warehousing products, and a host of other categories and subcategories of BI solutions, have been assisting organizations make better informed decisions for many years.

Business intelligence solutions are now being employed as hosted solutions and often with a portal interface, and are used increasingly for analysis of data collected from e-commerce and packaged application sources. Moreover, there is no end in sight. BI products are now in the midst of another evolution and repositioning, adding workforce planning, best-practices analysis, and business and revenue process optimization support, just to name a few. Equally important, the products will continue to become easier to employ, speaking the language of the end user and targeting additional domains of applicability.

Dan Kara is senior vice president and CTO of the Intermedia Group, a research and analyst firm based in Westborough, Mass. He is the chairman of Intermedia Group's Websourcing Conference and Exposition and Application Hosting and E-Service Forum. E-mail him at dkara@intmedgrp.com.

Software Magazine, October/November 2000, Copyright 2000