PAVING THE WAY TO DATA WAREHOUSE PROJECT SUCCESS: PART II
by Doug Laney

Consider the range of individual talents needed to deliver a data warehouse. You may need at least two distinct sets of technical talent -- one for each of the source systems and one for the target (data warehouse) system. You will need at least five sets of tool vendors: computing platform, DBMS, extract-transformation, end user query/reporting tool, and meta data management. Each will be needed at different project phases. Include both designers and developers for the data access, data acquisition, and meta data management construction and you've only just begun. Consider business analysts, data warehouse architects, computer operations, networking, training
development/delivery, marketing, and internal support - and now you've got a data warehouse development team!

Each of these sets of development personnel are needed at different points in the project and for different lengths of time. On successful data warehouse projects, sets of activities occur in concert, e.g. business requirements analysis, source systems analysis and data model analysis. However, project managers unfamiliar with building data warehouses often are inclined to take a conservative, serial "waterfall" approach to planning and scheduling for these distinct sets of talent. Unless parallel resource scheduling is done, even a well scoped iteration turns into a slow-paced 9 to 12 (or more) month effort.

Seasoned data warehouse project managers apply a parallel track approach to scheduling and managing talent to achieve the critical three-to-four month delivery timebox. It is customary for well-managed projects to comprise up to five parallel process tracks including:

1. project oriented activities focus on acclimating the company to data warehousing, aligning and managing the project team, implementing project training, establishing a support function, organizing internal marketing/communication, rolling out the data warehouse, and setting data warehouse strategic direction.
2. user oriented activities focus on understanding user data access/analysis requirements, modeling departmental data stores, designing and constructing the end user access environment, providing user training, managing the user acceptance process and evaluating the data warehouse.
3. data oriented activities focus on understanding existing operational data, modeling the atomic data warehouse structure, designing the data warehouse characteristics, creating the physical database, designing and constructing the data warehouse processing applications and populating and testing the data warehouse.
4. technically oriented activities focus on understanding existing or possible data sources, sizing the data warehouse environment, and determining, configuring, setting up and testing the computing environment.
5. meta data oriented activities focus on identifying existing sources of meta data, integrating technical and business meta data and designing/constructing a meta data delivery environment.

With parallel development planning and execution as part of the road map, data warehouse projects will travel short streets of success rather than long freeways toward failure.

Data Warehouse-Specific Approach

As important as the differences in the project management process -- iterative and parallel -- the clear architectural differences between a data warehouse and operational system warrant a need for work items specific to data warehousing.

As we have already discovered, traditional systems development methodologies at either end of the spectrum fly in the face of effective data warehouse project management techniques. On one end, traditional SDLC methodologies promote projects that are top heavy in analysis, and encourage addressing comprehensive sets of requirements over extended timeframes. On the other end, RAD methodologies promote the type of throw-away-and-do-it-again prototyping that is not conducive to projects that deal with the movement of gigabytes or terabytes of information.

Still, even if we were to tweak existing SDLC or RAD approaches to repair these shortcomings (as so many have attempted), major deficiencies, gaping holes, and erroneous techniques would remain. Aside from the fact that they would still not represent iterative or parallel processes, most of the techniques for architecting informational/analytical systems would be absent.

The following are a few examples of the many activities missing from these concocted "SDLC-DW" or "RAD/DSS" methodology medleys:

• no source system analysis activities to establish the cleanliness, completeness, accessibility and availability of candidate "systems of record"
• no data model analysis focused on selected subject areas to determine the existence of data to meet analytical requirements
• no data mapping, transformation design, and integration flow design
• no multi-level data architecting for atomic and aggregated data structures
• no design of data cyclicity, volatility, and multiple-granularity, etc.
• no data warehouse development for extract processing, data transport, data loading, post-load processing, etc.
• no meta data analysis, collection, integration and delivery

Using a roadmap expressly designed for developing data will help you avoid the potholes and dead ends of either a traditional, or a patchwork approach.

Data Warehousing "Best Practices"

When contemplating data warehouse projects, managers too often fall into the trap of focusing on which technology to purchase and which skills to rent. True, a set of data warehouse "best tools" and "best talent" can be expensive. But the decision to forego a data warehouse "best practices" road map to guide the project is the costliest decision a project manager can make.