EXCLUSIVE - PITT SC DIRECTORS SPEAK OUT
by Alan Beck, Editor-in-Chief
Interview with Mike Levine and Ralph Roskies, co-scientific directors of the
Pittsburgh Supercomputing Center...
HPCwire: LeMieux, your 3,000 processor HP AlphaServerSC terascale system, has
been at Pittsburgh Supercomputing Center (PSC) since October 2001 and in full
production mode, through the NSF PACI allocation process, since April 2002. It
remains, as it was when installed, the most powerful NSF system and that also
makes it the most powerful U.S. system available to all science and
engineering disciplines for unclassified research. What are the noteworthy
LEVINE: There have been excellent results across a range of application areas.
There's significant work in protein structure and dynamics. There's major work
in computational fluid dynamics of blood flow. In CFD, we've also enabled
simulations of new designs for power turbines and, related to that,
simulations of industrial-scale coal gasification. There's new work in solar
turbulence and magneto-convection. In other areas of physics, LeMieux has made
possible significant progress in quantum chromodynamics, also some
breakthrough work in photonics. There's also important work in cosmology and
major new findings in planet formation.
There's a lot going on, at many levels and with many approaches to using this
machine. LeMieux caught on rapidly. We regularly solicit feedback from
researchers, and a recurring theme has been "We couldn't have done these
computations without this machine."
HPCwire: Could you elaborate with a few examples?
ROSKIES: One of the earliest notable results came from Klaus Schulten and his
colleagues at the University of Illinois. They simulated a protein called
aquaporin, an important channel protein that regulates the passage of water
into and out of cells. The simulations solved a mystery that experiments
These were very detailed simulations, more than 100,000 atoms, and they gave a
precise picture of how the water molecules line up and move through this
channel. What they showed - which you couldn't see in experiments - is that
each water molecule does a 180 degree flip-flop at the midstream point of this
channel. Because of that flip-flop, protons can't come through, only the
Free passage of protons through this channel, which is ubiquitous in many
organisms, would totally disrupt the electrochemistry that drives metabolism.
Until these simulations, no one knew how the channel managed to keep the
protons out while letting water through. It was perplexing. Because he had
access to LeMieux, Schulten was able to get the answer and publish the results
Rita Colwell [director of NSF] highlighted this work, by the way, in her
keynote address at SC2002 in Baltimore.
Schulten's team was quick to take advantage of LeMieux because they developed
a molecular dynamics package, called NAMD, that's well adapted to large-scale
parallel systems. Using all 750 of LeMieux's nodes [each node has four
processors], they've sustained performance of over a teraflop per second. This
is for a real problem, not artificial benchmarking. It's outstanding and
probably sets a record for molecular dynamics, an application that generally
speaking doesn't parallelize well.
HPCwire: Can you mention one or two other notable results?
ROSKIES: Sure. Another paper in SCIENCE came from simulations of planet
formation -- by a team led by Thomas Quinn, an astrophysicist at the
University of Washington. In this case, the question was how so-called "gas
giant" planets form. These are huge planets, made mostly of gas, like Saturn
Quinn's objective was to model a new theory, based on gravitational
instability, of how these planets coalesce from the gaseous halo that swirls
around a young star. Because he had access to LeMieux, he was able to test
this model with high resolution, about 10 times better than anyone before. As
a result, he found that a gas giant planet can form in hundreds of years,
rather than millions as had been the standard thinking.
This was big news for many people who are now interested in planet formation.
In the past eight years or so, astronomers have for the first time found
planets outside our solar system, over 100 so far, and most of them are gas
Another fascinating project is the simulation of anti-microbial polymers by
Mike Klein and colleagues at the University of Pennsylvania. They used LeMieux
to show the feasibility of using a polymer to mimic anti-microbial peptides,
such as were first found on the skin of a frog. These natural peptides are an
important part of our defense against pathogens. If they can be synthesized,
we could use them to help defeat anti-biotic resistant bacteria, among other
things. They could also be used to create antibacterial clothing, bandages,
surgical instruments, a whole list of potential applications to address the
problem that hospitals are crawling with germs. One report last year said that
hospital-acquired infection is the fourth leading cause of death in the United
The big problem is that these natural peptides are extremely difficult and
expensive to synthesize. As Mike Klein says, you might as well be grinding up
diamonds. His group showed that a polymer structure, much easier to synthesize
than the natural peptides, would have virtually the same anti-microbial
properties. Lab work by his colleagues proved that this prediction was
correct. These results were published in the PROCEEDINGS OF THE NATIONAL
ACADEMY OF SCIENCE. This project was also one of six finalists in the 2003
Computerworld Honors program in Science. Because of this work, the University
of Pennsylvania filed for patents and created a company to exploit the medical
potential of these molecules.
LEVINE: We've also had some very large-scale simulations of earthquakes that
we should mention. This is a team of people led by Jacobo Bielak and Omar
Ghattas at Carnegie Mellon. They've used 2,048 processors with nearly 90
percent parallel efficiency.
They've sustained close to a teraflop per second performance over 12 hours.
This is an application that simulates soil conditions for the Los Angeles
earthquake basin. It's a very large-scale simulation that solves hundreds of
millions of differential equations for each time step in an unstructured mesh,
which subdivides the basin into more than 100 million grid points. This code
performs at nearly 25 percent of peak performance for the Alpha processor,
which is exceptional for an application like this, contrasted with synthetic
This project is directed toward getting better data on ground vibration so
that it's possible to create more rational building codes in earthquake prone
areas, such as Los Angeles and Mexico City. And these calculations, which are
a major step forward toward that goal, run at a scale -- 2,048 processors --
that simply is not feasible without a system like LeMieux.
Because of LeMieux, this group has also made significant progress in a
direction with wide implications for computational science -- a problem called
"the inverse problem." It's also sometimes called "blind deconvolution." Omar
Ghattas won the best paper award last year at SC 2003 for his parallel
inverse-problem algorithm. It's a sophisticated mathematical approach to
recover subsurface soil data from the observations on the surface.
This approach is extremely computationally intensive and ultimately will
require petaflop computing. But the potential is very significant, not only
for earthquake safety, but also for global climate change, and many other
areas -- ocean mapping, geological mapping. It's no surprise that DOE is
funding some of this work. In their most recent round of simulations, Ghattas
and Bielak have proven the concept for a model test case, in which they've
recovered subsurface soil parameters starting from surface seismic
observations. You need a LeMieux class system to do this.
Another application that just blazes on LeMieux is solid-state physics
software called LSMS, which we're using here in a couple different areas. This
software was originally developed at Oak Ridge National Lab, and one of our
staff scientists, Yang Wang, has worked extensively on adapting it to parallel
On LeMieux, using all 3,000 processors, LSMS has achieved 4.6 teraflops.
Obviously, this is an application that's very well adapted to parallel
Yang is collaborating with Mike Widom, a physicist at Carnegie Mellon, using
LSMS to simulate a solid-state phenomenon called amorphous metals -- to answer
questions about the unusual behavior of these materials. They are used, for
instance, for high-performance golf clubs, because they have unusual
springiness combined with strength and hardness.
HPCwire: What capabilities of LeMieux have been important in enabling this
LEVINE: Well of course right off there's the scale of the machine and the very
powerful Alpha processor. You have 3,000 of these processors all connected
together and that makes possible many large-scale simulations -- QCD is
another example -- that simply wouldn't happen without LeMieux.
Another key factor is the way these powerful processors are tied together. The
Quadrics interconnect network that we used gives us good latency and a very
high aggregate internal bandwidth. These are important factors in many
ROSKIES: It's worth noting also that it's relatively easy to port code from
other systems to LeMieux. Many researchers have told us this.
HPCwire: What have you learned from your experience with LeMieux that might
influence development of future systems?
ROSKIES: The main thing is the importance of memory bandwidth of the
processor. It's what we felt from the beginning. One of the reasons we chose
this processor is its superior memory bandwidth. I think we were right in that
LEVINE: Right. The bandwidth between nodes and to I/O devices is also
absolutely crucial. We made a very conscious effort to get the best out of
available technology, and we played an active role with Compaq in putting this
system together. This was not in any way a standard configuration we could
have just ordered from Compaq. It's also a machine of much larger scale than
any of this nature that had been fielded by Compaq. Scale matters and,
realizing this, we went to a great deal of difficulty to do it in the way that
made the scale our friend.
Likewise with the software -- I would say that it's a lesson for the future
that it pays to understand what your target is, what your target applications
are, what your target style of computing is, and to make changes to the
hardware configuration that are crucial to serve those goals. It's a lot of
effort, but important.
HPCwire: Compaq is now merged with HP, which happened after you installed
LeMieux. Does that have any impact on your future plans at PSC?
ROSKIES: If anything, the corporate commitment to high-end computing is
strengthened with this merger. HP's CEO, Carly Fiorina, made it a priority to
visit our site last year to take a first-hand look at LeMieux. She's very
interested in the scientific progress that's fostered through state-of-the-art
processing. She talked about this in her keynote speech at Comdex last
November. LeMieux is an accomplishment that HP rightfully takes some pride in,
that their technology contributes directly to solving problems like disease
LEVINE: We've been very impressed in our interaction with the people of HP
labs on both technical and scientific issues, and we think, in that sense, our
involvement with HP is going to help us go forward. The labs have excellent
experience with planning and layout of large data centers, for instance, and
they do detailed computational fluid dynamic simulations of airflow and
temperature in the machine room.
ROSKIES: They have a room that they use to reconfigure and experiment with
HPCwire: What impact has LeMieux had within PACI and NSF planning for
LEVINE: The delivery of this machine on time, on budget, and at the strength
and capability promised, has helped to put PACI on a solid footing from a
computational infrastructure point of view. From this foundation it has
continued to build. For NSF, the PACI program has represented a massive
scaling up in cyberinfrastructure. The first large piece of that puzzle, in
terms of a high-end system, was the Terascale Computing System -- LeMieux. It
has delivered the expected performance. In this sense, it's a keystone of this
program and it provides a firm footing from which the program can continue to
HPCwire: What are challenges of integrating LeMieux into the TeraGrid?
LEVINE: The TeraGrid initially was the PACI response to the NSF Distributed
Terascale Facility (DTF) solicitation. It was based on a homogeneous hardware
and software environment, that is Intel and Linux. The current challenge has
to do with the different operating system and software base that LeMieux
brings into the TeraGrid environment. LeMieux is the test case for
interoperability. This is a first pass at a grid environment incorporating
heterogeneity of system architectures. The challenge is to expand the software
suite to allow interoperability between different kinds of equipment.
The TeraGrid has established an Interoperability Working Group to coordinate
this effort, which initially is the effort of bringing LeMieux on board. Derek
Simmel of our staff is leading this Teragrid working group.
The point is not just to incorporate PSC and LeMieux, but also to establish
the necessary framework of interoperability so that the TeraGrid can expand
and other sites, other resources, can be seamlessly woven into this fabric.
What we're after is to create a sustained national cyberinfrastructure to
empower research and education.
ROSKIES: As part of this effort, we've established working relationships among
key staff people at PSC and their counterparts at the other TeraGrid major
resource centers, Illinois and San Diego, and also with Argonne and Caltech,
which are also TeraGrid sites. There are many technically challenging problems
that need to be solved to attain the goal -- to create the overarching look
and feel of homogeneity and ease of use of computation, data, visualization
and other services from the researcher point of view. This requires that the
partners work closely together. All of us are contributing our energies toward
this vision of national cyberinfrastructure, and it's coming along well.
HPCwire: In January, HP announced another server product line, code-named
Marvel, that extends the Alpha processor technology. PSC has installed some of
the first of these servers. How does this new system fit into PSC's plans?
ROSKIES: In February, we installed two separate, 16-processor Marvel systems,
each with 32 gigabytes of shared memory. The HP official HP product
designation for these servers is the GS1280. Each system represents an initial
phase of what will become two larger GS1280 systems, each composed of 128
processors and 512 gigabytes of shared memory.
There are two separate systems because one of them is funded by NIH and will
support biomedical work and the other is funded by NSF to support NSF science
and engineering. The NSF system will also be part of the TeraGrid.
These systems have a very fast processor, the EV7, 20 percent faster than the
EV68s of LeMieux. What's even more important, though, for these systems is the
memory structure. They're shared memory systems with an exceptional memory
bandwidth, the importance of which we've mentioned. The Marvel's memory
bandwidth is about six times greater than LeMieux.
The Marvels allow large shared-memory programming style, which is
complementary to the capabilities that we have in LeMieux, and they can
support a different and very important class of applications. We've identified
applications in genomics and structural biology and also visualization that
will benefit strongly from this architecture. Quantum chemistry is another
large application area that will run very well on the Marvels.
LEVINE: We've come up quickly with these systems -- they're already doing
production research by the way -- because they didn't come to us out of a
vacuum. We worked closely with HP in this process, and we prepared for the
February arrival of the first-phase production systems with extensive previous
work on early engineering test units and field-test machines. We and members
of the research community knew what we were getting well ahead of time.
HPCwire: HP, along with other vendors, appears to be committed to the Itanium
processor. How does this affect PSC?
LEVINE: I'm often asked about this, and the implication usually is that it
creates some sort of problem to change processors. But you have to keep in
mind that in high-performance computing we change processors all the time.
These transitions happen regularly, and if there's one thing we are expert in
at PSC it's in clearing out the underbrush to make a pathway with new systems.
Keep in mind that in about fifteen years we've debuted at least five
generations of new systems. We've evolved through the transition from Cray
vector systems to massive parallelism. With our last three lead systems, we
laid entirely new track, introducing systems to the research community.
There's no reason to think that switching from Alpha processors to something
else down the road represents something out of the ordinary. This is what we
ROSKIES: The Alpha EV7 and HP Marvel server is the best technology for at
least the next several years. Eventually that will change. In the meantime,
we're serving the research community in the best way we can. As new technology
emerges, we'll adopt it.
In this respect, we've already brought in, through our alliance with HP, a 32-
processor Itanium-2 cluster, which among other things helps in development and
testing of TeraGrid interoperability.
As with the Marvel, we're familiarizing ourselves with a new technology, with
significant lead-time for when it becomes a production system.
LEVINE: What we value from Alpha are things that have to do with its
electronic design plus the software suite. The design groups at HP that worked
on EV7 are now working on IA64, so you can expect a carryover of excellence,
with the difference that the Itanium is much more of a mass-market processor,
which brings the cost down.
The bright view of the future is that there will be carryover from the lessons
we've learned technologically from the Alpha plus the advantage of a broader
Of course hardware is only part of the picture. We're deeply invested in all
the grid-enabling efforts -- networking, storage, interoperability -- that are
essential parts of creating national cyberinfrastructure through TeraGrid.
This is a community effort. It's vital work and we're pleased to play a role.