"WAREHOUSE GALACTICA": BUILDING THE GLOBAL DATA WAREHOUSE
by John Ladley

Many Global 1000 enterprises with international LOBs or distributed operations are embarking on broad scope data warehouse projects. Project plans, user expectations and vendor relationships must all be adjusted to meet additional unique requirements.

A large percentage of companies' clients with global operations are using data warehouse technology (DWT) as a way to engineer data consolidation, quality, and analysis solutions that span lines of businesses (LOBs) and international borders. Much like Marshall McLuhan?s vision of the "Global Village" was brought about by the advent of modern mass communications, the "Global Data Warehouse" will also bring about profound changes in Information Logistics as the Internet dramatically reduces global boundaries. Additionally, DWT is being used to exploit opportunities inherent in common processes, such as purchasing and distribution.

Currently, organizations developing a global-aware data warehouse (DW) are overwhelmingly involved in international manufacturing or finance operations.

These industries present unique opportunities to leverage DWT with direct impact to business in the form of:

• Monitoring institutional performance -- Merging of external data with internal service and trading activity
• Financial consolidation -- Merging divisional and LOB performance to present uniform views of activity
• Currency fluctuations -- Managing difference in exchange rates by providing activity reports re-stated to reflect currency differences
• Distribution channel optimization -- determining effectiveness in light of freight, transport time and tariffs for supply chain management
• Procurement streamlining -- Optimizing purchasing by consolidating vendor lists to gain economies of scale

While the opportunities and returns to date are large, the barriers to successful use of global DWs can be equally large as well. Typical issues that are confronted by all DW projects become even more challenging in the face of global-aware DWs. These include but are not limited to:

• Database size -- International DW tend to grow even more quickly than average. Most start out with large fact tables , and it is not uncommon to see 50 - 100 million row tables
• Extensive transformations -- Language barriers and massively heterogeneous environments create extensive requirements to handle disparate formats of data
• Data usefulness issues -- Lack of standards at distributed facilities usually mean that the DW team receives data with all level of data hygiene requirements
• Political issues -- International manufacturing firms must often deal with autonomous locations that do not follow corporate policy or models. Additionally, these locations rarely allow direct access to "their" data. Autonomy creates political problems when shared data definitions are promoted. In many cases, we have noticed that there must be central metadata and local metadata to soothe problem users. Maintaining any type of architectural discipline require organizational muscle that many DW teams do not possess.

Moreover, additional specialized problems unique to the global nature for such DWs arise, such as:

• Configuration management -- Data collection from many countries requires consistent application of extraction logic. Additionally, there are problems with hardware and software consistency, as hardware can vary remarkably due to price and import restrictions.
• Time zone management -- Mission critical DWT usually requires bulletproof synchronization points of data. Multiple time zones need to be addressed in schedules and data storage.
• Multilingual, multicultural -- Most organization have a difficult time with one set of semantics. Managing multiple languages within a data model has frustrated many of these projects.
• Framework -- DWT structures in a global environment tend to be bi-directional. While the various plant and locations may provide data to a central DW, they also require some type of access to the centralized data or creation of a local data mart. In addition, local data that was not required for the central DW needs to be combined with DW data for local comparative purposes.

In general, Design, Process and Manage (DPM) tools, are being avoided or used solely in a highly specialized manner. This occurs because, despite growing feature/functionality, no single tool can cover the wide requirements, throughput and heterogeneity required of these environments. Constellar, for example, has recently demonstrated capabilities in this area.

Prism Solutions is also seen, although typically in a local or central version. We have seen several of large organizations develop their own data movement infrastructure based on scripts and internal data interchange procedures.

Data quality (DQ) control is an important issue, because remote locations can also view the new DW as a place to dump data that "may be" useful someday. Determining where to place the data filtering mechanism has proven to be a critical design issue. Cleansing at the source is usually difficult due to the aforementioned autonomy issues. Cleansing during a load process or after loading the DW can consume excessive cycles. Ideal implementations cleanse their data in a high performance staging area. Several of the data hygiene vendors feature international functionality (e.g., i.d.centric, ISI), however, the diversity of requirements usually requires the use of several DQ tools.

Data distribution back out to autonomous locations is difficult, given bandwidth, multi language requirements. Data mart requirements can vary widely, requiring the DW team to wither settle on a common denominator approach, or utilize data downloads that support local data mart or DW staffs.

Bottom Line: Global 2000 enterprises should carefully consider the processes required to develop and evolve international application of data warehouse technologies. Development teams must be given additional political pull to rally disparate sources of data around a common framework. Web-induced global "shrinking" will create high-return DWT scenarios for global companies that can put aside autonomy issues.

META Group expects Global 2000 size enterprises to wring leverage out of DWT through 1998 by applying technologies and business practices on a global scale. During 1999 / 2000, as Internet infrastructure matures into increasingly remote areas of the world, DWT will utilize the Web as the primary information access technology of choice for global data warehouse data movement, software distribution and data access. As DW becomes endemic to corporate application portfolios, further maturation of Web technologies will make global DW ubiquitous by 2000/2001, reducing the challenges faced currently by global enterprises.

---

For more information, see http://www.metagroup.com