DATA WAREHOUSES WILL TRANSITION TO INFORMATION DISTRIBUTION AND
COLLECTION CENTERS 11.11.97 by Michael P. Burwen D S *
Access to most data warehouses today is generally restricted to a relatively small number of so-called "knowledge-workers," who analyze the data contained in the warehouse in order to obtain information that will speed up or otherwise improve the business decision-making process. In a few years these same data warehouses, coupled with Internet and Intranet technology, will evolve into repositories of information that will be accessed by tens or hundreds of thousands of people representing an enterprise's employees, customers and vendors.
In a survey conducted as part of PAMG's syndicated multiclient study, Large-scale Database Solutions, a large sample of users reported the following:
Outside of the obvious requirements for lots of iron, network bandwidth, efficient software and other high-performance elements of the warehouse system, there are a number of issues that must be confronted if such a transition is to be successfully implemented. The five primary issues (in no particular order) are:
Security in such a system will become an increasingly thorny problem. Where hundreds of thousands, perhaps even millions of users are involved, the question of who is allowed access to what data at what times can get unbelievably complex. The number of permutations and combinations in some situations can seem nearly infinite, yet it all must be contained within a framework that is easily managed and and presented through an interface that doesn't take a rocket scientist to decode.
Such information distribution systems are as mission-critical to the enterprise as electronic funds transfer is to a bank or a reservation system is to an airline. Such systems must be up and running all of the time. One hundred thousand people twiddling their thumbs while the system is down is clearly not a tolerable situation.
By their nature, large, enterprise-wide information distribution systems are extraordinarily complex. While there are tools out there that can ease the pain (and many more will be forthcoming), there will not be any totally canned solutions that fit all needs (or even many) available in the foreseeable future. Today the shortage of computing and networking skills is well-known. Tomorrow, it will become much worse. How companies will obtain the necessary expertise to design, implement, manage and maintain such systems is a huge question mark that most will have to face. It is clear that outsourcing will become more prevalent, but there still will not be enough good people to go around.
Scalability has become the '90s slogan of virtually every software, hardware and network system vendor. It's easy to see how important scalability is given the growth rates mentioned in the preceding paragraphs. But scalability can't be taken by itself - it must be accompanied by performance. It may be that a system can scale to accommodate thousands of users, but if those users have to wait ten minutes for a response rather than the ten seconds it once took, the system may be essentially worthless.
Scalability and performance result from a long chain of interrelated components. These components - the processing and storage hardware, the operating system, the network components must all be well-matched in order to meet both scalability and performance requirements -- and these requirements must be met in the face of the 'First Law of Modern Data Processing'; i.e., there are never enough MIPS, bytes and/or bits/second.
In conclusion, the path that transitions today's data warehouses and data marts to tomorrow's information collection and distribution systems will not be an easy one. To get from here to there will require much long-term planning, not a strong point of many of today's shoot-from-the-hip, get-it-done-yesterday IT and business organizations. In addition, the implementation of such systems will require the application of significant financial and people resources, a tough recipe for many contemporary companies that are trying to be "lean and mean." Nevertheless, the long-run story is a compelling one. Companies that are first to implement such systems will enjoy both competitive advantages and substantial gains in productivity and efficiency.
The comments in this article derive from PAMG's study "Large-scale Database Solutions." For information about the study and its follow-on, "Data Solutions II," email datasol@pamg.com or check out the Web site http://www.pamg.com