SUN'S GUSTAFSON ON ENVISIONING HPC ROADMAPS FOR THE FUTURE
by Tim Curns, Editor
John Gustafson, a senior scientist and a principal investigator for Sun's HPCS
Program, took a brief departure from his work at Sun last year, refreshing his
"familiarity with the world of cluster computing and the needs of the HPC
community." He is now back at Sun, happily working with the 2009 timeframe of
the DARPA initiative. HPCwire tracked John down to examine some of his
thoughts on the hot topics in HPC today.
HPCwire: What do you find to be the most exciting technology right now?
John Gustafson: That's easy: Proximity Communication. Breakthroughs like this
don't come along very often. Here we thought we'd be doomed to worsening the
balance between compute speed and communication speed, and have to continue to
burden programmers with awareness of data locality. Thanks to Drost's and
Sutherland's innovation, we can realize almost three orders of magnitude more
bandwidth in and out of chips. Who would have guessed that we'd ever be able
to build a tightly-coupled supercomputer again? And it looks like it might not
just be a reprieve, but a real solution, since now we can expect that the
bandwidth improvements will track Moore's law just like device densities do.
HPCwire: What important technologies do you see as emerging in the next few
JG: Well, let's start with hardware advances. Optical interconnect will get
much more practical for short-distance communications. Chips will apply
asynchronous design principles, both to let each part of the chip run as fast
as it can and to save power. I foresee a return to liquid cooling in the next
few years, because air cooling is hitting its limits at many large HPC sites.
Of course we'll be putting multiple processor cores on one chip, and multiple
threads per core, which is going to mess with the minds of people who insist
on knowing how many "processors" there are in a system.
By the way, I predict you'll see some dramatic improvements to the way we do
floating-point math in the next few years. We won't just be throwing 64-bit
IEEE 754 at every problem and hoping for the best. We'll use sometimes more
precision (because we'll discover we need it) and sometimes less (because too
much precision wastes bandwidth and memory and electrical power). We'll have
better tools for assuring numerical validity. You may know that Sun already
offers Interval Arithmetic in its compilers, and it continues to pick up
momentum in the HPC community. That's one promising approach, but certainly
not the only one. We need better math.
For software advances, we can thank DARPA for a renewed emphasis on ease-of-
programming. In the last decade, we lost ground in programmer productivity,
but soon we should get it back and then some. HPC programmers have long been
presented an ugly choice between ease-of-use and performance. Now it looks
like we might be able to have it both ways: high-level expressions like those
of Matlab or Mathematica, but higher performance than if you wrote the
operations out in low-level C or Fortran and used an optimizing compiler. Only
when systems run _faster_ when programmed with high-level abstractions will
HPC programmers use those abstractions. I know, because I'm one of those
There's an advance coming that may be just as important to the future of HPC
as anything I just mentioned: meaningful benchmarks. There was a time when
LINPACK was a good way to motivate vendors to work on their floating-point
performance, but now it's doing a disservice because it can motivate vendors
to ignore memory system requirements. The HPC Challenge Benchmarks and the
Purpose-Based Benchmarks show a lot of hope for establishing rankings that
correlate with real workloads. But it will take years to wean some people off
their Top 500 addiction and get them asking the right questions about system
HPCwire: How will Sun be involved in leveraging these technologies?
JG: Every single thing I just mentioned is being used in our HPCS approach.
For some of the technologies, like Proximity Communication, we're developing
the intellectual property entirely in-house and protecting that IP carefully.
The Purpose-Based Benchmarks are developed at Sun but shared freely with
vendors and customers. And for things like optical communication, we work
closely with third parties that are leaders in their areas to make sure
they're targeting HPC-specific needs.
HPCwire: What about parallel processing? Does that fit under the umbrella of
Grid computing? What does Grid computing mean to you, as definitions of it
defy general consensus?
JG: There are indeed some areas where parallelism is so easy that there's no
problem farming it out over a geographically distributed set of machines.
Factoring large integers, or processing SETI data, for example. But for
physical simulations, dispersing processors geographically for simultaneous
execution makes no sense to me. Supercomputing is currently limited by heat,
size, and cost. If you make the distance greater, you'll use more total
electricity (though the waste heat will at least be less concentrated). The
longer communication wires will cost more than if things were closer together.
So the way to use the Grid to supercomputing is simply to use it to tap into a
closely-knit cluster or SMP that offers the lowest cost for the compute
You wouldn't believe the effort our engineering team puts into removing every
inch from the interprocessor communications path. Adding miles is
inconceivable in the realm of HPC applications we're addressing.
I hear very different uses of the term "Grid computing." It usually refers to
the ability to get to computing on a network, like an electric power consumer
uses a power grid. I think you can probably imagine our CEO, Scott McNealy,
cracking a lopsided grin and saying, "Gosh, what a novel idea!" I first heard
the idea of utility computing expressed by a guy named Ross Perot in the early
1980s. Sun has pitched this as the way to go since the company was founded in
1982, so the idea goes back at least that far.
The big advantages of the Grid are that it gives you the competitive pricing
of many choices, and it balances the load so you make better use of all the
systems that are out there. I hope people start using quality-of-service
measures in their Grid pricing, particularly as it applies to floating-point
arithmetic. For instance, if you do Interval Arithmetic on different
computers, your quality is how narrow the interval is that includes the
answer. Right now, if you do HPC-type calculations on different computers on
the Grid and get different floating-point answers, you have no way of knowing
which one to trust.
That's really a big hurdle that we should work to overcome. My vote would be
to expand the set of applications that we can handle with Interval Arithmetic,
and that solves the problem of porting approximate answers over the Grid. But
if people have other suggestions, I'm listening!
HPCwire: While we await responses to that, let's look at some of your history.
Could you explain your work with the Atanasoff-Berry Computer (ABC) and its
restoration? Why is this important?
JG: We are going from petascale to, uh, milli-scale! OK, back to computing in
One reason I think historical computers are so important is because they often
made brilliant use of the parts and switching technology they had at the time.
There's a general sense that because computing technology changes so quickly,
we have nothing to learn from work that's more than ten years old. That's
absolutely not true. Engineers keep reinventing concepts and giving them new
names, unaware that a little knowledge of prior art could have saved them the
The ABC was the first electronic computer to use base-2 math, and to make the
giant leap of using Boolean logic to perform arithmetic electronically. But
it's also the first machine to use dynamic RAM... capacitors, periodically
refreshed to maintain their voltage indicating a zero or one value. That was a
clever way to get the cost down, since capacitors cost pennies and vacuum
tubes cost dollars back then. And the ABC had a system clock: 60 Hz from the
line voltage! Moore's law works quite well if you go backward from Moore's
1970 paper, incidentally. In 1939, the ABC had 0.3 kilobytes of RAM in about
two cubic feet of space.
The original ABC was dismantled, and the only original parts remaining are a
memory drum and one of the binary add-subtract units. People get very
emotional about claims of who was "first" to do something in early computing,
and the ENIAC defenders are exceptionally unwilling to admit that their
"first" was based on any prior art. That was one reason I got involved.
The ABC was designed to solve dense linear equations. See, even back in the
1930s people wanted a LINPACK solver! Solving linear systems was the killer
app, because they could limp along through many things with pencil-and-paper
and Marchand or Monroe mechanical calculators. I suggested to Jack Dongarra
that maybe we could get the ABC on his list, with a speed of 0.06 flops per
second. Not megaflops. Flops. It took about 15 seconds to do a multiply-add!
Maybe we could start a "Bottom 500" list, with the ABC as the all-time
champion. Kidding. But if you like plotting graphs of LINPACK speed through
the years, you can now do it going back 65 years! Quite a baseline.
HPCwire: It sure is. How did you get involved in such a project?
JG: I did my Ph.D. work at Iowa State, and used to walk every day through the
physics building where the ABC was invented. There was a poster on the wall
bragging about it, and I always thought, "Yeah, right. I'll bet every
university in the country thinks they invented the electronic computer." I was
a typical cynical grad student.
You may recall that Sun acquired the part of Cray that used to be Floating
Point Systems. When I was at FPS in the 1980s, I managed the development of a
machine called the FPS-164/MAX, where MAX stood for Matrix Algebra
Accelerator. It was a general scientific computer with special-purpose
hardware optimized for matrix multiplication (hence, dense matrix factoring as
well). One of our field analysts, a well-read guy named Ed Borasky, pointed
out to me that our architecture had precedent in this machine developed a long
time ago in Ames, Iowa. He showed me a collection of original papers reprinted
by Brian Randell, and when I read Atanasoff's monograph I just about fell off
my chair. It was a SIMD architecture, with 30 multiply-add units operating in
parallel. The FPS-164/MAX used 31 multiply-add units, made with Weitek parts
that were about a billion times faster than vacuum tubes, but the
architectural similarity was uncanny. It gave me a new respect for historical
computers, and Atanasoff's work in particular. And I realized I shouldn't have
been such a cynic about the historical display at Iowa State.
I also learned about the controversy regarding the ENIAC, how Mauchly had
appropriated Atanasoff's ideas as his own, in what may be the greatest act of
intellectual piracy in the 20th century, and that hit a nerve with me.
The ENIAC clan claimed that the ABC never worked, or was never finished to the
point where it could do anything useful. So when I heard that George Strawn
and Del Bluhm were planning to get funds to reconstruct the ABC from the plans
and other data, I jumped in with both feet and ultimately led the project to
completion. And guess what? The ABC works very well, thank you! It solves
systems of linear equations. Very, very slowly. Like, eight hours for 29
equations in 29 unknowns. And we discovered that the original machine had been
used for solving least squares problems in the Iowa State Statistics
Department, so it wasn't as if it never became a "production" computer.
HPCwire: What about your immortal code? Describe your work here as well,
please. Does it relate to your current work?
JG: "Immortal code" was a phrase I coined for an effort to write computer
programs that could survive at least 50 years of changes in technology.
Mathematics doesn't go out of date; why should algorithms? The problem has
been that even the most mathematically-expressed algorithms still make all
kinds of assumptions about the machinery they'll run on. The classic programs
assumed sequential execution, and that arithmetic was much harder to do than
moving data around. We all know how those assumptions have been invalidated by
technology, so now we have to rewrite tons of code that had seemed very solid.
How do we prevent this from happening again?
We discovered that a lot of the "software rot" out there resulted from people
stepping out of their proper role, and doing someone else's job badly. Like a
chemist who unrolls a loop in the belief that it helps the compiler, only to
later discover the loop runs faster in its original and easier-to-read form.
Or the system software designer who assumes physicists want an alarm to sound
and the program to halt when e to a negative big number causes an underflow.
The solution was for each player -- hardware designer, system software
designer, application programmer, user -- to stay strictly in role and
communicate explicitly what happens at their interfaces. Think of it as the
communication of both requirements and concessions, with an emphasis on
concessions. If you can tell a library designer, for example, that you don't
care if intermediate results in complex arithmetic cause overflow, then the
library can promise much higher speed in return. About the only way we have
now of communicating these things is with clunky compiler flags that are
arcane and vendor- specific.
It's funny that you should ask about this, because a lot of that work turned
out to be exactly what DARPA needed in its HPCS program. HPCS requires
robustness and portability, and the Immortal Code work gave me a head start in
working out how we might be able to write software in such a way that it
doesn't have to be reverse-engineered and modified at huge expense every few
years, like we do now.
HPCwire: Ok, that brings us back to HPCS. Could you describe Sun's HPCS
research programs that are related to the DARPA grants? What kind of
advancements are being made here? What phase are you in now? How are you
JG: There are, at last count, 24 teams working on the HPCS program at Sun,
covering every aspect of the system. Administration, I/O, Proximity
Communication, Purpose-Based Benchmarks... and Product Concept Definition.
Like IBM and Cray, Sun is in the middle of "Phase II" right now, which is the
minimization of the risks in the radical ideas we proposed in Phase I. The
idea is that by the end of Phase II, we'll have a plausible case for actually
building a system using what we've learned, and can propose a real product to
DARPA that would have broad commercial and government use.
At SC2004, we really let people have a good look at how Proximity
Communication allows us to build the equivalent of Wafer-Scale Integration,
but with high yield and repairability. We'll be dribbling out other milestones
over the next few months, so watch for our announcements. There's a lot more
where this came from.
As for my role, I got Sun to Phase II and was PI for the program initially.
I'm now concentrating on the applications of the Hero system. I'm one of the
dyed-in-the-wool HPC developers at Sun and as such I can infuse real-world
experience into the design process. I provide in-house guidance about what
does and what doesn't matter to the HPC community.
HPCwire: What DOES matter to the HPC community? What's next in this area,
particularly for Sun?
JG: We're getting incredible response to our "utility computing" program...
and virtually all the early adopters are in HPC. Manufacturing. The energy
industry. Life sciences. A lot of Sun's revenue comes from technical and
scientific sales. But you have to realize that you're talking to Sun's biggest
HPCwire: Of course. Will Sun be involved in further government initiatives?
JG: The HPCS program is certainly not the first collaborative development
project Sun has done jointly with the government... and we have several
interesting projects underway currently. I can tell you that we're doing what
it takes to make a case for Sun being in Phase III of HPCS, and that we have
excellent support from Sun executives.
Why Sun? I hear that question at conferences a lot. People had this impression
that Sun only builds workstations, then they had the impression that we only
sold dot-com hardware, so why are we involved in HPC? That's a crazy
pigeonhole to put us in, like saying that IBM only builds mainframes. The
thing Sun can now bring to HPC is the full range of memory models, from cheap
commodity clusters to giant flat-address-space SMPs, and offer the
architecture that actually matches the application. If a customer has an
application that works with message passing, then Sun has Opteron-based
clusters that have outstanding price/performance. But if they don't have
locality in their application, or don't want to invest in the programming
effort to fit the MPI model, then fine. We've got an SMP roadmap that goes to
2010 and beyond and scales to huge numbers of processor threads.
HPCwire: How has HPC become even more important in the last few years? Where
do you see HPC heading in the next 5 years?
JG: HPC is always important, even if some other markets temporarily steal all
the limelight now and then. You need "world's fastest computer" efforts to
make ALL of computing make advances... that's what's been happening for many
decades. The breakthroughs come from the bleeding edge stuff, and then
percolate down to more conventional computing.
In the last few years, there's been increasing realization that clusters are
very cost-effective, accompanied by a realization that they don't cover all of
the applications we used to do with supercomputers. So the main trend I see
for HPC is a huge effort to fill in that gap. One way is to incrementally
improve cluster software and put in a lot of work to make them turnkey
application servers. Another way is to architect monolithic, closely-coupled
systems that are so much easier to use that the saving in programmer and
administrator cost is more than enough to offset the higher hardware cost.
You'll see a lot of both approaches in the next few years.
HPCwire: Thanks so much for taking the time to answer some questions about
your work and HPC in general. I'm sure the community will be watching your
work with Sun very closely.
John Gustafson has been published many times and has served as a computational
scientist, SCL associate scientist, adjunct associate professor of Computer
Science and Applied Mathematics at Iowa State University, subject area editor
for Journal of Parallel and Distributed Computing, and on the editorial board
for Parallel and Distributed Computing Practices.