Repurposing Supercomputers - What happens on The Other Side?

project designed to determine the feasibility of us­ing re-purposed supercomputer hardware for research purposes. Gary Grider, Division Leader for High Performance Computing at LANL came up with the idea for PRObE in 2006 after arriving to the conclu­sion that many of their computer systems that are normally decom­missioned and subsequently destroyed despite still having quite a bit of useful life left in them. Many facilities deal with their decommissioned systems by putting them on trucks and driving them to a secure facility where the components are placed in an industrial metal shredder which chops them into tiny pieces which are then melted down to re­cover precious metals. But does something that might have cost $30M just three or four years prior really only possess scrap value today? Neither Grider or Jacobson thought so and co-wrote the NSF proposal together other collaborators from Carnegie Mellon University and the University of Utah. In October 2010 the NMC was awarded $10M from the NSF to build PRObE.

From a pure profitability standpoint the answer to the scrap value question is probably yes. Based on historical trends it is usually possible to achieve about double the performance in a 10th of the floor footprint and ⅔ to one half of the power consumption by performing an upgrade of systems that are four years into production. As we will see, the operational expenses (OPEX) for running an outdated computer system quickly exceeds the capital expense (CAPEX) investment with the accompanying reduced OPEX for a new, more efficient system.

Many universities that begin deploying cluster style research computing often resort to using discarded desktop computers. However, these cobbled together systems are simply not adequate to meet the needs of researchers who require very large computer systems to perform their research. This means the value of a decommissioned supercomputer might be significantly higher than the scrap value to the average person or researcher at a university because these older systems can provide plentiful and more powerful computational capabilities than would otherwise be available.

A Different Approach

PRObE is an answer to getting these decommissioned systems into the hands of people who can use them, but setting up and maintaining large clusters containing more than 1000 nodes requires overcoming several obstacles:

1) Sheer volume: Decommissioning, moving, inspecting, troubleshooting, and bringing back thousands of old computers on­line takes sig­nificant time and effort. Also, unlike when a system is slated for de­struction - care must be taken through­out the decommissioning pro­cess so that parts are not damaged.

2) Space: A computer system with 1000 or more nodes and appropriate interconnect networks will likely require about 40- 50 whole racks of computer equipment. PRObE has capacity for 1MW of compute power, about 280 tons of cooling, and 3000 sq ft of server room space to house these large machines. This is sufficient for housing two large and a few smaller clusters.

3) Electricity cost: 1MW costs around $1M per year in New Mexico. It is a required OPEX and in PRObE’s case, is provided by NSF funding. This is not a typical setup, but since there is no procurement cost for the computers - the electricity is covered instead. This allows PRObE to provide the compute services to the community at no cost to the individual users.

4) Lack of spare parts: Vendors do not necessarily keep old spare parts around once a product has reached end-of-life and sometimes the vendor of an old system might have vanished. In such cases, the only outlet is the gray market - such as eBay and other vendors specializing in reused computer equipment. In PRObE’s case - LANL’s systems are usually larger than what PRObE can house, so a sufficient number of spares (typically about 20 percent) can accompany each system. Machines can also be cannibalized to keep the system running once the spares run out.

5) Staff to operate: PRObE is successful primarily because of the workforce we use to build the clusters and to maintain them. In particular, our staff is creative as they can both assemble and maintain the hardware even with limited funds. Instead of hiring consultants or full time staff members to perform this work, PRObE relies on local high-school and early college talent, which is also a wonderful way to train young people. Over the past 6 years we have employed close to 40 high school students that spend a couple of hours with us each week. During summers and winter break, these students often work full time. To PRObE this is an affordable solution and the students get hands-on experience building large computer systems.

The Future

PRObE is fortunate that the NSF sees the value in what we do, the training that we provide, and the scientific value these older systems can contribute to the academic and scientific communities. Without NSF support, PRObE would not be possible. While the operation of PRObE require both skill and creativity, the work is rewarding and the scientific benefits are as real as exemplified by the many research citations PRObE regularly receives in the scientific literature.