Features - Enterprise Data Insights:

SANs: BEWARE OF HALF-TRUTHS
by Gian Jagai, Storage Consultant, GlassHouse Technologies Inc

"Beware of the half truth. You may have gotten hold of the wrong half." -- Unknown

Your company has decided to take the plunge and invest in a storage area network (SAN). The benefits seem like a dream come true: better performance, greater uptime, increased storage utilization, ease of management, data replication and disaster recovery options. A SAN will allow your employees to remove their firefighting hats and put on their thinking caps and begin to proactively manage the environment. Customer satisfaction will increase as a result and the cost of maintaining the environment will fall dramatically. Indeed you will achieve nirvana, at least that's what storage vendors all over will tell you.

However, the truth is a lot less cosmic. Yes, it is true that while you can indeed achieve some of the promise of a SAN, you will undoubtedly be confronted with new problems that you were not aware of when first presented with nirvana. Therein lies the truth about SANs: unforeseen problems are inevitable. In fact most companies will spend countless hours and a great deal of effort doing research, finding the 'right' vendor and getting the 'best' price, yet still experience challenges. While this is a necessary process, greater consideration should be given to what happens once the environment is deployed and in production.

Imagine yourself in the following scenario: After months of vendor evaluations, design reviews and dealing with project management headaches, you have a functioning SAN in production. Confidence is high and expectations are higher. After the vendor's implementation consultants leave, you and your staff are left to become experts overnight on the new SAN. Sure, you can hire the vendor to stick around for a couple months after initial deployment to help as your staff grows more comfortable with the environment. However, this reduces one of the potential benefits of a SAN, "reduced TCO." Well, maybe you have some smart folks on your staff, or they might have had prior experience with SANs. Great, now you're reliant on one or two key individuals to maintain the SAN. They must also provide support for the SAN, provision storage for new systems and of course continue to perform their pre-SAN duties. There go a couple more SAN benefits: ease of management, customer satisfaction and proactive maintenance -- not to mention overworking the best members of your team.

Alright, so you accept you will need to hire additional personnel dedicated to running the SAN. Fast forward about a year after the initial deployment. In that time, your SAN has grown in capacity and number of attached hosts and spread to other production environments. By now you and your group are cruising along; you work hard but are keeping up with requests for more storage. Performance is excellent and uptime has increased. Utilization is not where you would like it but you believe that will improve in time. However, a serious problem affecting the availability of your storage has presented itself. The solution, a new firmware update to your storage arrays, is recommended by your vendor. No problem, your arrays can perform online non-disruptive upgrades and they can be done during non-production hours. While that may be true, the vendor also mentions that the new firmware has not been regression tested for compatibility with your older versions of switch firmware, host based multi-pathing software and HBA drivers and firmware. Failure to perform any of these upgrades means that the vendor cannot continue to provide you with technical support.

Instantly you are presented with a project much larger and delicate than the initial deployment, as you now have live customers with production data running in many different departments, some in 24x7 environments. It is unlikely that you could buy your way out onto another storage platform, since this SAN was supposed to solve your storage needs for a couple years. If you forgo vendor support then you lose more potential benefits: increased performance, uptime and higher customer satisfaction. You have no choice but to start a new project performing the upgrades. Nirvana has suddenly become hell, and you are in charge of it all.

One company presented with a similar situation used the following approach to solving this complex problem:

• Inventory all components dependent on the array firmware. This can include Host OS, HBA drivers and firmware, version of multi-pathing software, switch firmware, and serial numbers.
• Place all new components (drivers, firmware, multi-pathing software versions, and controllers) in a single accessible repository.
• Test the vendor supplied upgrade procedures in a lab environment.
• Customize vendor provided upgrade procedures for the IT staff, making the instructions specific to the environment.
• Determine upgrade parameters, including impact to availability of systems and maintenance window required. Create a matrix to show all of the different variants currently in production.
• Identify local IT staff responsible for performing upgrades in each geographic location.
• Train local IT staff on procedures.
• Identify lowest risk installations on which to perform initial production upgrades.
• Schedule maintenance windows with users.
• If a vendor is doing part of the upgrade, review the process, the number of times they have done it, any previous problems and the escalation process with them.

As you can see the promise of a SAN can be enticing but the reality can easily become a nightmare. Companies looking to implement SANs should be conscious that once deployed they quickly become intermingled into a company's most important systems. Also, be aware that not all vendors perform the same levels of regression testing on their equipment. This of course is usually reflected in the price you pay for the system. Also, not all SANs require an equal amount of work to maintain. Some vendors, through their production and engineering methods, will design storage hardware that is inherently easier to manage and maintain than others.

When thinking about the workload associated with SANs you can expect your staff to spend 33% of their time implementing additional deployments or reconfiguring storage to meet new growth and changing user requirements; another 33%, troubleshooting problems and errors which may or may not be associated with the SAN; and the remainder of their time reviewing new technology. This would include evaluating new firmware, drivers, management software and new hardware enhancements. Thus some hidden costs related to the SAN include management time, upgradeability and limitations of the vendor support matrix. These costs can also vary greatly depending on the storage vendor.

As the example above shows, upgrading components in an existing environment requires more time and involves more complexity than the initial deployment, representing a hidden cost associated with maintaining the SAN. Enlisting the expertise of an independent storage consultant, who can act as your advocate through the vendor selection process, see through a vendor's sales pitch, design a complete solution, and point out possible pit-falls would be invaluable. Additionally, a good consulting team will provide overall project management, allowing you to focus on meeting business and customer demands while keeping you in touch with the entire process. Such a resource can help you to reach nirvana and get there without a hang-over.

Gian Jagai is a storage consultant at GlassHouse Technologies. Gian has been in the storage industry for more than 5 years and has extensive experience in SAN design, implementation and storage assessments. GlassHouse, a leading vendor-independent provider of storage services and consulting, is headquartered in Framingham, MA. gjagai@yahoo.com or gjagai@glasshouse.com