ANALYTICAL MODELING OF SYSTEM CAPACITY AND PERFORMANCE

Executive Summary

This white paper positions the benefits of using analytical modeling as a viable alternative to physical benchmarking when sizing enterprise server upgrades. It cities a specific positive customer experience with Merisel, in an instance where both server sizing approaches were used. The benefits and accuracy of the overall analytical modeling approach are discussed in detail.

Program Overview

The use of various configuration sizing techniques has been widespread since the consummation of the very first computer system sale. The selection of the appropriate CPU, memory and I/O components is of paramount importance to the ultimate success of any IT transaction. Vendors have made large investments in ensuring that customers are offered the correct equipment to meet their needs, both current and anticipated. Purchasers need the full confidence that their selections will deliver upon expectations.

Hewlett Packard has consistently refined its methodology for the sizing and configuration of new systems and add-on components. The approach taken for making the recommendations falls into two broad categories: the first is to actually assemble the proposed component(s) and to create a physical environment that almost exactly mirrors the intended configuration, the second is to create an accurate representation of the environment through the use of sophisticated analytical modeling techniques.

This whitepaper focuses on HP's use of analytical modeling techniques to accurately derive custom configuration recommendations. In order to pragmatically illustrate the merits and value associated with such an approach the experiences of a real-world HP client, Merisel Inc, have been cited.

Hewlett Packard's Capacity Planning Center (CPC)

HP's Capacity Planning Center (CPC) based in Cupertino, California is one of the custodians of the expertise and knowledge that the company has amassed over decades of conducting benchmarking, configuration and performance management activities. The center is home to many performance specialists acknowledged throughout the company as leaders in the field of performance consulting and benchmarking.

The CPC has been in existence for over 17 years and during this period has performed thousands of benchmarks, covering the full spectrum of HP's server processor product lines. Non-disclosure agreements prohibit the amassing of a library of results, however, the knowledge gained from each encounter is preserved and shared among the CPC engineers as well as with other HP divisions.

The CPC's charter is to provide the HP End-User and Channel sales force with high quality support in analytical modeling, benchmarks, proof-of- concept, and performance escalations. These services allow the Sales Representatives to eliminate performance and proof-of-concept as issues in the sales process.

The program currently offers over 32 highly trained engineers with an average tenure of over five years in the CPC. Skill sets range from Project Management to detailed root cause analysis, from organizational and scheduling skills to hardware trouble-shooting. The CPC engineers are widely known within HP and the customer community for their expertise and professionalism in performance consulting.

The CPC conducts over 700 activities a year that typically fall into one or more of the following categories, benchmarking, consulting, and analytical modeling.

Benchmarking

Benchmarking is an approach in which a live customer environment is reproduced in the CPC, specifically for the purpose of measuring the performance of the customer's applications on HP hardware. It can include a full complement of processors, memory, discs, network components, applications and a high-tech simulation of actual users to replicate the application loading of a real-life, real-time environment. Benchmarking is obviously very resource intensive given its requirements of the dedicated systems and analysts physically present to perform the tuning, monitoring and measurements.

A typical benchmark takes a week to set up and configure the hardware environment, and one or more weeks to conduct the tests and do the tuning. Some tests take over a month to complete (this is determined by the complexity of the tests and the amount of data being processed). Benchmarking allows HP customers to see first-hand how their environment and their own live data would work with HP products. Plus, it affords the opportunity to try various scenarios for best performance according to that customer's specific requirements. One important aspect of benchmarking is to have an environment as the CPC Lab that is free of unknown variables and influence. Such an environment is isolated and controlled: the hardware, network and peripherals.

A derivative of the full benchmark involves the system under test being driven by one or more systems called drivers, which play a set of predetermined scripts. The elimination of live users (and their associated terminals/clients) from the test reduces the overall complexity of the configuration and adds consistency to the test itself. Add to that the burden of generating a realistic script. A benefit, however, is that tests can be repeated whenever desired, independent of typical human constraints! Script creation can take one to two man-weeks depending on the complexity and volume of unique transactions included in the benchmark suite.

Analytical Modeling

Analytical modeling can be an excellent alternative to benchmarking. It offers very good accuracy, but requires a far more modest investment and, additionally, benefits from dramatically improved speed of execution. As the name suggests, analytical modeling is the science of creating a logical model that accurately replicates the behavior of a target environment. A sophisticated set of algorithms is created and iteratively tuned to a level that permits a multitude of configuration and transaction loading scenarios to be faithfully modeled. In order to effectively profile the characteristics of an application suite(s) it is necessary to have an understanding of behavior. The CPC will provide instructions on how to set up the data collection for custom or complex environments to facilitate the modeling activity. Data collection takes place at a clients facility and can be conducted on any platform that is using the software in a manner similar to that of the intended new environment.

There are certain instances where analytical modeling offers benefits beyond the use of standard benchmarking. An analytic model has the ability of projecting the performance of systems that cannot be benchmarked. For example, when a new system has just been announced, but not yet available for benchmarking, an analytical model can be utilized to effectively project its performance in targeted environments.

Other uses of modeling include performance planning. As an example, a customer may be experiencing growth rates of 10% per month and need to know when an upgrade will be required. In today's business climate, it is common for one company to merge or acquire another company. Analytic modeling can combine workloads from different systems and project the combined performance of both workloads running on a single system. When a benchmark is required, analytic modeling can be used to offer additional data. The benchmark may be used to establish a baseline for performance characteristics that can b used to derive a robust model.From this model performance profiles can be projected for a variety of configuration and workload levels.

The Analytical Modeling Approach in Detail

Analytical modeling is a powerful tool that can offer accurate performance analysis at a fraction of the cost of a benchmark. Analytical models are mathematical representations of a particular computer system. Queuing theory is used to define the relationships between various resources (e.g. CPU, disks, memory, etc.) and their queues. These algorithms are populated (parameterized) using measurements taken from a running system. Once the model is built, parameters can be changed to represent possible changes to the running system. The model can accurately project the impact of these changes.

There are many uses for analytic models. Most uses revolve around "What If" type questions, such as "what if I double the number of users on my system?" A model can be built to address this question in a matter of a few days compared with a benchmark that traditionally may require weeks or even months.

What the Process Looks Like

All analytical modeling studies are typically based on the following four step process:

1. Define and document the specific objectives of the modeling study.
2. Obtain relevant data, either from collection on a system with similar workload profile or using archived data from prior study. This step typically includes: Installation/configuration of measurement software Monitoring of the suitability of collected data Submittal of data to Capacity Planning Center
3. Analysis of submitted data
4. Presentation of findings

In more detail:

Step One: Define and document the specific objectives of the modeling study. Typically a brief statement is created that outlines the required outcomes from the projections; in certain cases these can be very detailed. An example would be: "what system configuration will be required to run my current system next year if we grow 50%?" A more detailed example would be: "what size system will be required if my ABC application grows 100% and my 'XYZ application grows 25%?"

Step Two: Obtain relevant data. The sophistication of the modeling tools utilized by the CPC has dramatically increased over the course of the last decade. However, the use of a common data collection source over this period of time has provided rock steady consistency. The collection software is designed to be minimally invasive and implementation does not require a reboot of the target system.

Configuration of the collector software is conducted in two phases. First, the correct data types to be collected are identified. Specification of the data types being recorded is done by the insertion of a simple command string in a control file. Secondly, each application is categorized to map to the study's objectives as stated in step #1. To illustrate; if the object of the study is to model the growth rate of a specific application a definition is made of the exact components that constitute that particular application group. The specification of the application groups referenced in the modeling objectives frequently requires detailed knowledge of applications in order to accurately identify the active elements that need to be incorporated in the finalized analytical model. This particular step of the overall process benefits greatly from the accumulated in-depth experiences of the CPC personnel and their expertise in partnering with the customer's detailed application knowledge.

In many instances there is only one opportunity for collecting a specific set of performance data. For example, transactions resulting from a year-end close or during a particular high-transaction period are classified as nonrepeatable events. For this reason clients and their HP account teams are advised to submit sample data prior to the critical monitoring period to ensure that the tools are installed and functioning correctly.

While certain aspects of the process are fully automated, it is important to capture and provide the CPC analysts with descriptive narrative relating to general perceptions of the workload running on the system and any further circumstances that could have direct impact on the study during the monitoring period. These could include peak periods, average periods, periods where users may not all be present (birthday, holiday, unexpected weather impacting business, etc). These factors are taken into consideration during the final analysis of the environment and the model's projected behaviors.

Data recorded in the collection files typically covers an extended period of time. The CPC analysts will narrow down the period under scrutiny to identify specific occurrences where potential configuration bottlenecks could occur. The selected timeslices will contain the actual data ultimately used to construct the analytical model.

Step Three: Analysis of submitted data. Recorded data is sent to the performance analyst for modeling. This information will include the original modeling objectives, the actual collection logfiles, narrative related to the environment and possibly an indication of time periods suitable for constructing the base model.

CPC analysts then review the measurement data, build the base model, calibrate and validate the model, and then alter the actual configurations and workloads to project the anticipated workloads. For a basic model, this step will require 4 to 5 working days of an analyst's time and will result in a summary report. The report will minimally include a summary of the projections that will address the modeling objective.

Step Four: Presentation of findings. This step may or may not involve the performance analyst. It is frequently delivered by a Technical Consultant or other performance knowledgeable field resource. Another potential delivery option can include the distribution of copies of the report and the scheduling of a conference call between the CPC analyst and all interested parties, with the analyst facilitating a focused discussion session.

Analytical Modeling - The Tools

The tool uses an analytic modeling approach based on open and closed queuing models to project performance. The tool collects specific information on configurations and workloads. Configuration information encompasses the exact specification of the hardware environment including the CPU, memory, and disks. Workload information is the measurement data collected from the system under test and defines the work that is being applied to that system. The actual analytical model is a collection of parameters and algorithms that define the behavior of a computer system.

There are many different approaches to implementing a capacity-planning model. The favored toolsets offer flexibility in being capable of taking multiple approaches. A Mean Value Analysis (MVA) approach is offered that solves models using a heuristic iterative approach. This iterative approach converges to a solution very quickly. A discrete-event simulation approach is also available which introduces a simulation time attribute to the MVA approach.

Both modeling approaches are based on the Central Server Model that was originally introduced in the mid-1970's. This approach has been revised in several ways to more accurately reflect the 1990's style of computing, to increase accuracy and to optimize the model definition and execution.

The Central Server Model was based on the concept of a single, centralized, system. All terminals were of extremely limited intelligence and all processing was conducted by the single system. By comparison, the exponential increase in the complexity and sophistication of today's systems (multiple networked processors, high-powered clients, multi- tiered distributed applications, etc.) dictates a more updated approach.

There are several additional and noteworthy differences between the current model and the 70's Central Server Model. First, today's model is an "open model". An open model means that there may be an infinite number of arrivals over any given time period, that is, the arrivals are not bounded. Once a transaction leaves the system, it is gone and there is no correlation with the next arrival. This is in contrast to the "closed model" where transactions arrive from terminals via a delay center. Transactions wait or delay a specified amount of time (the "think time") and then enter the system. The total number of terminals limits the number of transactions in the system. The current tool is capable of solving both open and closed model formats.

The Merisel Situation

Headquartered in El Segundo, California, Merisel Inc is a North American $4.6 Billion company, distributing a full line of 25,000 computer hardware and software products from the industry's leading manufacturers to resellers throughout North America. A complete range of customized value-added services complements the product sales. Merisel also offers a wide suite of dedicated support services to high-end resellers through its Merisel Open Computing Alliance and the company's Enterprise Computing Division.

As part of the company's ongoing commitment to customer satisfaction, HP's vice president of sales commissioned a benchmark for Merisel. Merisel had been an entrenched user of IBM mainframes and the proposed HP solution included technologies that had not yet been introduced. In this case the recommended configuration centered around the HP 9000 V-Class Enterprise Server, a model with no publicly released TPC or SAP performance metrics available.

Merisel planned to run their entire operation on two instances of SAP one in Canada and the other in the USA. The Canadian implementation was fully operational and all attention was placed on the vender selection decision in the United States.

The challenges were summarized by Merisel's senior vice president of Information Technology, Mary West, "One of the biggest issues that we had was with performance. What we found is that the type of business that Merisel runs drives very high transaction volumes through SAP with the need for sub-second response time. Our US Company is in the range of 4-5 times bigger than our Canadian business, we were really focused on making sure we had sufficient capacity to meet both our volume, transaction and response time requirements."

Coupling the stated emphasis on performance related criteria, with the unfeasibility of assembling a representative V-Class system, populated with a full-complement of peripherals and applications, the obvious move was to turn to an analytical modeling based approach.

The Merisel Experience

Merisel's Director of Computing, John Buckley, commented: "We started the process in November '97 where we looked at a series of vendors and potential solutions - at the time the V-Class was just coming out, so we relied upon an analytical benchmark. With the HP team we put together a set of forecasts of what we believed to be US volumes and then we applied those forecasts - this enabled the HP team to come back with a V-Class recommendation. The V-Class analytical model really provided us with the background to make the decision - we selected the V-Class over any other vendor."

The configuration emulated in the model was a HP 9000 V-Class Enterprise Server with 4 CPUs, 4GB of memory running Oracle and SAP. HP PCs were used as the drivers.

Models were created from data collected at Merisel on October 7, 1998. The models were validated and then used to project performance as the workload increased from the recorded level of 56,250 dialog boxes per hour, through an increase of 250%. The measurement data showed that the database server, from which the data was recorded, was efficiently tuned with good disk balancing. In addition, the environment scaled well as the workload increased and CPU capacity was added.

The model projections showed that a HP 9000 V2200 with 4 processors would support a workload increase of about 100% to around 110,000 dialog boxes per hour. This level was short of the 2,000,000 dialog boxes per eight-hour day goal or 250,000 per hour. An upgrade to a V2200 with 10 processors would provide the CPU capacity needed, plus additional upgrades could be made to the server, including adding more CPUs to handle significant growth beyond the 2,000,000 dialog boxes per eight-hour day.

The rigor and discipline surrounding the creation and application of the analytical model for the environment gave the Merisel IT Management team the fullest confidence in their decision to purchase the correct V-Class system. The application suite resident on this system is business-critical to the Company.

The use of a comprehensive analytical model permitted Merisel to select exactly the right system configuration for their business. Waiting for the VClass processor to become available, and to construct the appropriate benchmarking environment in order to conduct a benchmark, would have injected a significant time delay in the whole selection and purchasing process. The analytical modeling approach enabled Merisel to rapidly reach their decision and for their whole company to benefit from the power of the new system. In a world where the old cliché of time being money has never been more true, analytical modeling delivers the goods.

The CPCs constant pursuit of excellence resulted in another benefit for Merisel. In order to continually check and refine their understanding of the accuracy of the analytical models they create, the CPC will sometimes elect to conduct a full physical benchmark to measure the system's actual performance against that projected by the model. The final comparison confirmed the total accuracy of the original analytical model.

For details about the specific Merisel tests, please contact your HP Sales Rep who will be able to get the information from the CPC.

This was committed to Merisel and performed once the system was fully commissioned and operational.

Summary

About 9% of the CPC current activities involve modeling. Most sales situations call for benchmarks, either because a customer insists on 'seeing the activity or modeling is not a good option for their situation. Customers who are currently utilizing other vendors equipment may not be able to produce the data necessary to conduct a model. Some clients do not have the time or expertise to perform the required data collection, others do not have available resources to devote to it, while others may not be testing growth but just want a proof-of-concept type of benchmark. Additionally, modeling requires specific skills that reside with a handful of CPC engineers, thus resources for this activity occasionally get constrained.

In general, modeling offers an accurate, less expensive alternative to benchmarking. The results generated are proven to be highly accurate and can be derived in a dramatically more expeditious manner than traditional benchmarking.