The Beowulf Cluster is one of the reasons I jumped onto the Linux bandwagon in the late 1990s. The idea of making good use of old computers to solve real world problems fascinated me. As computers continue to shrink in size, price, and power consumption, the reuse aspect is less appealing than before, but the concept still draws me in from time to time. I only wish I could justify the cost of building one.
A Beowulf Cluster generally refers to a collection of consumer-grade computers connected by a local network that run parallel-processing applications. The typical Beowulf Cluster runs on the Linux OS and distributes processing tasks using either Message Passing Interface (MPI) or Parallel Virtual Machine (PVM) libraries.
- Wikipedia articles for Beaowulf Cluster, MPI, and PVM
- Beowulf.org FAQ and mailing list archives (1997-current)
- Some notes on how to build a Linux cluster by Guillaume and/or Jennifer Dargaud, 2003
How To Build A Cluster
Building the cluster is half the battle (the other half is writing programs for it). Thankfully, there are a lot of how-to pages out there. Some cover just the basic steps while others provide more explanation and background information. It’s been interesting to watch them evolve over time to, in length, complexity, and style. Here is a sample:
- Building Your Own Beowulf Cluster by David H. M. Spector, Wired, 2000
- Building Linux Beowulf Clusters by Mike Perry, fscked.org, 2000
- How to Build a Beowulf Linux Cluster The Mississippi Center for Supercomputing Research, 2001
- Sun™ Based Beowulf Cluster by Börje Lindh, Sun Microsystems, 2002
- The Beowulf HOWTO by Kurt Swendson, 2004
- Engineering a Beowulf-style Compute Cluster by Robert G. Brown, Duke University, 2004
- Building a Beowulf System by Jan Lindheim, Caltech, 2005 (last entry on Internet Archive)
- Beowulf Cluster Design and Setup by Amit Jain, Boise State University, 2006
- Building a Beowulf Cluster in just 13 steps by Sanath Kumar, Linux.com, 2009
- Building a simple Beowulf cluster with Ubuntu by Serrano Pereira, University of York, 2013
Finally, this isn’t a how-to article, but it’s worth the read. Cluster Urban Legends: Build Your Cluster With Facts Not Fiction, written by Dr. Douglas Eadline in 2007, debunks some of the major myths and misunderstandings surround Beowulf Clusters. In short, it helps you determine if you are really building one for the right reasons.
So, you’ve built a Beowulf Cluster. Now what? Well, now You have to write programs for your cluster to solve all of the world’s complex problems. Beowulf Clusters are built for crunching numbers, not serving up web pages. That means they are used primarily in the sciences, though I can think of a few business applications that could benefit from the extra processing power (think derivatives pricing).
I have a lot more research to do in this area. Currently, I have only a link to some information about an old text book to offer. I need to add more tutorials to this list, especially in the use of MPI and PVM, and how to determine when it’s appropriate to use one over the other. Check back later or watch my Twitter feed for updates.
- Designing and Building Parallel Programs by Ian Foster, Addison-Wesley, 1995
Real World Beowulf Clusters
Years ago, I started compiling a list of real Beowulf Clusters that had been built for various purposes. I still have the list, though some of the links are now gone or only available on the Wayback Machine Internet Archive. I was hoping to do a little write-up on each cluster on the list, but if that doesn’t pan out, then will just add the list in this section.
And of course, most scientists, engineers, and computer geeks are gifted with a great sense of humor (even if we are the only ones that understand our own jokes).