Features:

SUN'S GUSTAFSON ON ENVISIONING HPC ROADMAPS FOR THE FUTURE
by Tim Curns, Editor

John Gustafson, a senior scientist and a principal investigator for Sun's HPCS Program, took a brief departure from his work at Sun last year, refreshing his "familiarity with the world of cluster computing and the needs of the HPC community." He is now back at Sun, happily working with the 2009 timeframe of the DARPA initiative. HPCwire tracked John down to examine some of his thoughts on the hot topics in HPC today.

HPCwire: What do you find to be the most exciting technology right now?

John Gustafson: That's easy: Proximity Communication. Breakthroughs like this don't come along very often. Here we thought we'd be doomed to worsening the balance between compute speed and communication speed, and have to continue to burden programmers with awareness of data locality. Thanks to Drost's and Sutherland's innovation, we can realize almost three orders of magnitude more bandwidth in and out of chips. Who would have guessed that we'd ever be able to build a tightly-coupled supercomputer again? And it looks like it might not just be a reprieve, but a real solution, since now we can expect that the bandwidth improvements will track Moore's law just like device densities do.

HPCwire: What important technologies do you see as emerging in the next few years?

JG: Well, let's start with hardware advances. Optical interconnect will get much more practical for short-distance communications. Chips will apply asynchronous design principles, both to let each part of the chip run as fast as it can and to save power. I foresee a return to liquid cooling in the next few years, because air cooling is hitting its limits at many large HPC sites.

Of course we'll be putting multiple processor cores on one chip, and multiple threads per core, which is going to mess with the minds of people who insist on knowing how many "processors" there are in a system.

By the way, I predict you'll see some dramatic improvements to the way we do floating-point math in the next few years. We won't just be throwing 64-bit IEEE 754 at every problem and hoping for the best. We'll use sometimes more precision (because we'll discover we need it) and sometimes less (because too much precision wastes bandwidth and memory and electrical power). We'll have better tools for assuring numerical validity. You may know that Sun already offers Interval Arithmetic in its compilers, and it continues to pick up momentum in the HPC community. That's one promising approach, but certainly not the only one. We need better math.

For software advances, we can thank DARPA for a renewed emphasis on ease-of- programming. In the last decade, we lost ground in programmer productivity, but soon we should get it back and then some. HPC programmers have long been presented an ugly choice between ease-of-use and performance. Now it looks like we might be able to have it both ways: high-level expressions like those of Matlab or Mathematica, but higher performance than if you wrote the operations out in low-level C or Fortran and used an optimizing compiler. Only when systems run _faster_ when programmed with high-level abstractions will HPC programmers use those abstractions. I know, because I'm one of those programmers!

There's an advance coming that may be just as important to the future of HPC as anything I just mentioned: meaningful benchmarks. There was a time when LINPACK was a good way to motivate vendors to work on their floating-point performance, but now it's doing a disservice because it can motivate vendors to ignore memory system requirements. The HPC Challenge Benchmarks and the Purpose-Based Benchmarks show a lot of hope for establishing rankings that correlate with real workloads. But it will take years to wean some people off their Top 500 addiction and get them asking the right questions about system performance.

HPCwire: How will Sun be involved in leveraging these technologies?

JG: Every single thing I just mentioned is being used in our HPCS approach. For some of the technologies, like Proximity Communication, we're developing the intellectual property entirely in-house and protecting that IP carefully. The Purpose-Based Benchmarks are developed at Sun but shared freely with vendors and customers. And for things like optical communication, we work closely with third parties that are leaders in their areas to make sure they're targeting HPC-specific needs.

HPCwire: What about parallel processing? Does that fit under the umbrella of Grid computing? What does Grid computing mean to you, as definitions of it defy general consensus?

JG: There are indeed some areas where parallelism is so easy that there's no problem farming it out over a geographically distributed set of machines. Factoring large integers, or processing SETI data, for example. But for physical simulations, dispersing processors geographically for simultaneous execution makes no sense to me. Supercomputing is currently limited by heat, size, and cost. If you make the distance greater, you'll use more total electricity (though the waste heat will at least be less concentrated). The longer communication wires will cost more than if things were closer together. So the way to use the Grid to supercomputing is simply to use it to tap into a closely-knit cluster or SMP that offers the lowest cost for the compute cycles.

You wouldn't believe the effort our engineering team puts into removing every inch from the interprocessor communications path. Adding miles is inconceivable in the realm of HPC applications we're addressing.

I hear very different uses of the term "Grid computing." It usually refers to the ability to get to computing on a network, like an electric power consumer uses a power grid. I think you can probably imagine our CEO, Scott McNealy, cracking a lopsided grin and saying, "Gosh, what a novel idea!" I first heard the idea of utility computing expressed by a guy named Ross Perot in the early 1980s. Sun has pitched this as the way to go since the company was founded in 1982, so the idea goes back at least that far.

The big advantages of the Grid are that it gives you the competitive pricing of many choices, and it balances the load so you make better use of all the systems that are out there. I hope people start using quality-of-service measures in their Grid pricing, particularly as it applies to floating-point arithmetic. For instance, if you do Interval Arithmetic on different computers, your quality is how narrow the interval is that includes the answer. Right now, if you do HPC-type calculations on different computers on the Grid and get different floating-point answers, you have no way of knowing which one to trust.

That's really a big hurdle that we should work to overcome. My vote would be to expand the set of applications that we can handle with Interval Arithmetic, and that solves the problem of porting approximate answers over the Grid. But if people have other suggestions, I'm listening!

HPCwire: While we await responses to that, let's look at some of your history. Could you explain your work with the Atanasoff-Berry Computer (ABC) and its restoration? Why is this important?

JG: We are going from petascale to, uh, milli-scale! OK, back to computing in the 1930s.

One reason I think historical computers are so important is because they often made brilliant use of the parts and switching technology they had at the time. There's a general sense that because computing technology changes so quickly, we have nothing to learn from work that's more than ten years old. That's absolutely not true. Engineers keep reinventing concepts and giving them new names, unaware that a little knowledge of prior art could have saved them the trouble.

The ABC was the first electronic computer to use base-2 math, and to make the giant leap of using Boolean logic to perform arithmetic electronically. But it's also the first machine to use dynamic RAM... capacitors, periodically refreshed to maintain their voltage indicating a zero or one value. That was a clever way to get the cost down, since capacitors cost pennies and vacuum tubes cost dollars back then. And the ABC had a system clock: 60 Hz from the line voltage! Moore's law works quite well if you go backward from Moore's 1970 paper, incidentally. In 1939, the ABC had 0.3 kilobytes of RAM in about two cubic feet of space.

The original ABC was dismantled, and the only original parts remaining are a memory drum and one of the binary add-subtract units. People get very emotional about claims of who was "first" to do something in early computing, and the ENIAC defenders are exceptionally unwilling to admit that their "first" was based on any prior art. That was one reason I got involved.

The ABC was designed to solve dense linear equations. See, even back in the 1930s people wanted a LINPACK solver! Solving linear systems was the killer app, because they could limp along through many things with pencil-and-paper and Marchand or Monroe mechanical calculators. I suggested to Jack Dongarra that maybe we could get the ABC on his list, with a speed of 0.06 flops per second. Not megaflops. Flops. It took about 15 seconds to do a multiply-add! Maybe we could start a "Bottom 500" list, with the ABC as the all-time champion. Kidding. But if you like plotting graphs of LINPACK speed through the years, you can now do it going back 65 years! Quite a baseline.

HPCwire: It sure is. How did you get involved in such a project?

JG: I did my Ph.D. work at Iowa State, and used to walk every day through the physics building where the ABC was invented. There was a poster on the wall bragging about it, and I always thought, "Yeah, right. I'll bet every university in the country thinks they invented the electronic computer." I was a typical cynical grad student.

You may recall that Sun acquired the part of Cray that used to be Floating Point Systems. When I was at FPS in the 1980s, I managed the development of a machine called the FPS-164/MAX, where MAX stood for Matrix Algebra Accelerator. It was a general scientific computer with special-purpose hardware optimized for matrix multiplication (hence, dense matrix factoring as well). One of our field analysts, a well-read guy named Ed Borasky, pointed out to me that our architecture had precedent in this machine developed a long time ago in Ames, Iowa. He showed me a collection of original papers reprinted by Brian Randell, and when I read Atanasoff's monograph I just about fell off my chair. It was a SIMD architecture, with 30 multiply-add units operating in parallel. The FPS-164/MAX used 31 multiply-add units, made with Weitek parts that were about a billion times faster than vacuum tubes, but the architectural similarity was uncanny. It gave me a new respect for historical computers, and Atanasoff's work in particular. And I realized I shouldn't have been such a cynic about the historical display at Iowa State.

I also learned about the controversy regarding the ENIAC, how Mauchly had appropriated Atanasoff's ideas as his own, in what may be the greatest act of intellectual piracy in the 20th century, and that hit a nerve with me.

The ENIAC clan claimed that the ABC never worked, or was never finished to the point where it could do anything useful. So when I heard that George Strawn and Del Bluhm were planning to get funds to reconstruct the ABC from the plans and other data, I jumped in with both feet and ultimately led the project to completion. And guess what? The ABC works very well, thank you! It solves systems of linear equations. Very, very slowly. Like, eight hours for 29 equations in 29 unknowns. And we discovered that the original machine had been used for solving least squares problems in the Iowa State Statistics Department, so it wasn't as if it never became a "production" computer.

HPCwire: What about your immortal code? Describe your work here as well, please. Does it relate to your current work?

JG: "Immortal code" was a phrase I coined for an effort to write computer programs that could survive at least 50 years of changes in technology. Mathematics doesn't go out of date; why should algorithms? The problem has been that even the most mathematically-expressed algorithms still make all kinds of assumptions about the machinery they'll run on. The classic programs assumed sequential execution, and that arithmetic was much harder to do than moving data around. We all know how those assumptions have been invalidated by technology, so now we have to rewrite tons of code that had seemed very solid. How do we prevent this from happening again?

We discovered that a lot of the "software rot" out there resulted from people stepping out of their proper role, and doing someone else's job badly. Like a chemist who unrolls a loop in the belief that it helps the compiler, only to later discover the loop runs faster in its original and easier-to-read form. Or the system software designer who assumes physicists want an alarm to sound and the program to halt when e to a negative big number causes an underflow. The solution was for each player -- hardware designer, system software designer, application programmer, user -- to stay strictly in role and communicate explicitly what happens at their interfaces. Think of it as the communication of both requirements and concessions, with an emphasis on concessions. If you can tell a library designer, for example, that you don't care if intermediate results in complex arithmetic cause overflow, then the library can promise much higher speed in return. About the only way we have now of communicating these things is with clunky compiler flags that are arcane and vendor- specific.

It's funny that you should ask about this, because a lot of that work turned out to be exactly what DARPA needed in its HPCS program. HPCS requires robustness and portability, and the Immortal Code work gave me a head start in working out how we might be able to write software in such a way that it doesn't have to be reverse-engineered and modified at huge expense every few years, like we do now.

HPCwire: Ok, that brings us back to HPCS. Could you describe Sun's HPCS research programs that are related to the DARPA grants? What kind of advancements are being made here? What phase are you in now? How are you involved?

JG: There are, at last count, 24 teams working on the HPCS program at Sun, covering every aspect of the system. Administration, I/O, Proximity Communication, Purpose-Based Benchmarks... and Product Concept Definition.

Like IBM and Cray, Sun is in the middle of "Phase II" right now, which is the minimization of the risks in the radical ideas we proposed in Phase I. The idea is that by the end of Phase II, we'll have a plausible case for actually building a system using what we've learned, and can propose a real product to DARPA that would have broad commercial and government use.

At SC2004, we really let people have a good look at how Proximity Communication allows us to build the equivalent of Wafer-Scale Integration, but with high yield and repairability. We'll be dribbling out other milestones over the next few months, so watch for our announcements. There's a lot more where this came from.

As for my role, I got Sun to Phase II and was PI for the program initially. I'm now concentrating on the applications of the Hero system. I'm one of the dyed-in-the-wool HPC developers at Sun and as such I can infuse real-world experience into the design process. I provide in-house guidance about what does and what doesn't matter to the HPC community.

HPCwire: What DOES matter to the HPC community? What's next in this area, particularly for Sun?

JG: We're getting incredible response to our "utility computing" program... and virtually all the early adopters are in HPC. Manufacturing. The energy industry. Life sciences. A lot of Sun's revenue comes from technical and scientific sales. But you have to realize that you're talking to Sun's biggest HPC bigot.

HPCwire: Of course. Will Sun be involved in further government initiatives? Why Sun?

JG: The HPCS program is certainly not the first collaborative development project Sun has done jointly with the government... and we have several interesting projects underway currently. I can tell you that we're doing what it takes to make a case for Sun being in Phase III of HPCS, and that we have excellent support from Sun executives.

Why Sun? I hear that question at conferences a lot. People had this impression that Sun only builds workstations, then they had the impression that we only sold dot-com hardware, so why are we involved in HPC? That's a crazy pigeonhole to put us in, like saying that IBM only builds mainframes. The thing Sun can now bring to HPC is the full range of memory models, from cheap commodity clusters to giant flat-address-space SMPs, and offer the architecture that actually matches the application. If a customer has an application that works with message passing, then Sun has Opteron-based clusters that have outstanding price/performance. But if they don't have locality in their application, or don't want to invest in the programming effort to fit the MPI model, then fine. We've got an SMP roadmap that goes to 2010 and beyond and scales to huge numbers of processor threads.

HPCwire: How has HPC become even more important in the last few years? Where do you see HPC heading in the next 5 years?

JG: HPC is always important, even if some other markets temporarily steal all the limelight now and then. You need "world's fastest computer" efforts to make ALL of computing make advances... that's what's been happening for many decades. The breakthroughs come from the bleeding edge stuff, and then percolate down to more conventional computing.

In the last few years, there's been increasing realization that clusters are very cost-effective, accompanied by a realization that they don't cover all of the applications we used to do with supercomputers. So the main trend I see for HPC is a huge effort to fill in that gap. One way is to incrementally improve cluster software and put in a lot of work to make them turnkey application servers. Another way is to architect monolithic, closely-coupled systems that are so much easier to use that the saving in programmer and administrator cost is more than enough to offset the higher hardware cost. You'll see a lot of both approaches in the next few years.

HPCwire: Thanks so much for taking the time to answer some questions about your work and HPC in general. I'm sure the community will be watching your work with Sun very closely.

John Gustafson has been published many times and has served as a computational scientist, SCL associate scientist, adjunct associate professor of Computer Science and Applied Mathematics at Iowa State University, subject area editor for Journal of Parallel and Distributed Computing, and on the editorial board for Parallel and Distributed Computing Practices.