A very recent (actually, currently ongoing) suite of benchmark results of a dual Athlon. Note: This site has been temporarily shut down pending "permission" from AMD to open it again, since they were apparently concerned about numbers being published from a pre-release chipset. I technically doubt that I need their permission to publish (especially with all the caveats I had clearly posted about the the numbers being "beta" numbers from a "beta" chipset:-), but am happy enough to postpone it as a matter of courtesy. Check back in a few weeks (from 3/16/01).
A more advanced introduction to beowulfery (focusing more on parallel program development and profiling is rgb's Extreme Linux tutorial presented at the 1999 Linux Expo. Available both as a Postscript Image and a PDF Image
The more advanced introduction to beowulfery continues (focusing more on hardware benchmarking and design) in rgb's Extreme Linux presentation at the 2000 Atlanta Linux Showcase and Conference. Available both as a Postscript Image and a PDF Image The paper associated with the talk (published in the conference proceedings by USENIX) is also available in online format or postscript or PDF but be warned that there are more and niftier figures in the talk than in the paper.
A very basic introduction to beowulfery for the lay-person, presented to the 2000 Techno-Security conference (for law enforcement personnel from small local police forces through the FBI, and from several countries) held in North Myrtle Beach from April 17-19, 2000. The applications discussed were primarily decryption and pattern matching/predictive modeling. Postscript Image and a PDF Image.
A draft snapshot of my current manuscript of a book on beowulfery. This is a work in progress and certain to change significantly. Please send mail to rgb with comments, criticisms, and suggestions. Although not a GPL work (I have fantasies of a real paper book) it is available during its development time for private and personal use by anyone on the beowulf list. Please don't (re)publish it. Available both as aPostscript Image and a PDF Image.
rshbench is small shell script and binary package that can be used to test and tune remote shell performance. It is primarily useful for clusters, distributed computing, and beowulfs, but might be of use to a systems administrator trying to assess the additional cost of e.g. pam or inetd/tcpd on top of/in addition to ssh or rsh.
and as a
and finally as a
memtest can be used to reveal streaming and random access memory limits that can form bottlenecks. It can be used to test swap as well. Not very sophisticated, be warned.
cpu_rate is a timing harness and microbenchmark program. It measures "bogomegarates", of a variety of sorts, where the "bogo" stands for bogus -- they are not to be taken too seriously. Still, the tool can easily reveal the effect of L1 and L2 cache sizes on floating point arithmetic rates for simple vectors. The tool wraps a variety of interesting measures of numerical or memory access speed -- +-*/, +-*, stream (with dynamic memory allocation and command line selectable stride), the savage transcendental benchmark, read/write speed of a vector with access sequential forward, sequential backward, or shuffled random, the speed of /dev/random, and the time required for an empty subroutine call. More tests may be added in any snapshot, and with the tarball below it is pretty easy to add your own.
This is all GPL v2b (b for beverage, read the man page or copyright notice:-)
There is a short summary of CPU performance as a function of vector size (and comments) in a CPU Vector Performance Summary based on a much earlier version of cpu_rate on this website.
The following vendors are for the post part not really endorsed. Some of them I have direct experience with. Others have just contacted me and asked to be added to the list, and I've done so (without even charging them, which is probably slightly insane on my part). If they're really and truly quasi-dedicated linux/beowulfish vendors I really have no objection, BUT...
...USE THEM AT YOUR OWN RISK. I make no warranty whatsoever about any given vendor being suitable, cheap, or reliable unless I've used them before or they've bribed me heavily with tee shirts, coffee mugs, free palm pilots, or (always useful) money. Of course if they've bribed me I'll say anything they want about how great they are, so you shouldn't trust even an endorsement.
Intrex is a the local vendor "down the road" that provides very reasonable systems and parts (literally) over the counter, at least to folks in North Carolina. Driven strictly by demand, they've started to put together very reasonably priced rackmount systems, where they'll basically build you exactly what you want at OTC prices. I recently (10/00) got a quote from them for rackmount 2U nodes -- 800 MHz PIII's with 256 MB PC133 ECC, a 10 GB HD and floppy, and a NIC per node for just about exactly $1K/node. This would be even cheaper now as SDRAM drops towards $0.50/MB from over a dollar at the time of the quote. Their "integration charge" per node is basically nothing -- maybe $50 -- as they make their primary profit on the volume of the sale, just as they should.
I like these guys and do business with them pretty regularly. They are the sole supplier for the systems in my home beowulf because they are quite cheap and literally two miles away from my house. This allows me to get service "instantly". For example (true story), 12/4/00 I was rebuilding my home beowulf (putting it on a nifty heavy duty steel and particle board shelf unit, actually, to which I had attached rolling casters -- very cool and cheap!) and naturally my primary server/desktop (a dual Celeron on the Abit BP6) refused to come back up -- no power at all. I suspected a blown power supply, but didn't want to mess with the swap game at home.
So, I simply picked up the unit and carried it to the car, and trundled off to their South Square store. I plopped it down on the counter and said "I think the power supply is blown". Twenty five minutes later I was out with a new power supply installed at no charge and no questions asked, as the system was still less than a year old (by around three weeks:-). Of course I also grabbed some newly cheap SDRAM and even popped for a 30 GB disk for all my MP3's while I was there...
This kind of service is why I buy from local vendors if/when I can find a decent one. Intrex is exceptional even in this regard -- their systems come with a lifetime labor warranty, and that's the expensive part of buying ANY system -- hardware (replacement or otherwise) is cheap, but the time required to swap power supplies, motherboards, and so forth around to identify and replace a bad part is very, very expensive. To me -- it is my time. I'll therefore make an exception for Intrex and "endorse" them even though they don't give me coffee mugs or mouse pads for free. Their prices are low enough I can afford to buy my own with what I save.
ASL Computers is a vendor of linux (only!) boxes of all sorts, from desktops through large rackmount servers. They sell both 1U and 2U units and I imagine will do systems integration sorts of things for you. I don't know if they consider themselves a true "turnkey beowulf vendor" (I'd generally reserve this term for companies with a true beowulf expert on staff and on the beowulf list who can help with more than just the hardware configuration and initial software install) but they can certainly sell you cluster components with linux preloaded. If they have or establish an outsourcing relationship with e.g. Scyld, they might even qualify without an on-staff expert...
These guys get some kudos from me as they provided the access and close support to the fabled early-release Dual Athlon benchmarked elsewhere on these pages. While I cannot personally attest to the quality of their hardware (never having bought any) I can say that their systems guys seem quite competent, which means that their preinstalled linux boxes likely work just fine. No tee shirt(s), but hey, the chance to play with a Dual Athlon before they even officially exist makes up for it;-)
Linux NetworX (formerly known as Alta Tech) is yet another linux/beowulf turnkey vendor. They sent me some very cool tee shirts with these centipedes on the front. They are worth a call if you are looking for quotes on a ready-to-run cluster system at pretty much any level of performance (up to the very high end). They also have some very pretty node-mounting boxen (not exactly racks -- custom boxes -- which is what I first knew them for as Alta Tech). Here is a synopsis of their business that they contributed (note that I did not write this, they did:-):
"Linux NetworX brings its powerful cluster technology to those demanding high-availability and high-performance systems. With the use of cluster computer technology, Linux NetworX provides solutions for companies with high-computing needs including Internet servers, research, industry, government and other technological fields. Through innovative hardware, complete cluster management software, service and support, Linux NetworX provides end-to-end clustering solutions. To date, the company has built some of the largest cluster systems in the world and has developed unique hardware and versatile software to facilitate overall system management. Linux NetworX has offices in Utah, New York, Calif. and Texas and worldwide distributors."
"About ClusterWorX: ClusterWorX allows users of Linux NetworX clusters to control the cluster as a single system, and provides remote monitoring and management capabilities. ClusterWorX can be accessed through an easy-to-use graphical user interface (GUI), command line and HTML. Other management tools include remote access, disk cloning and serial access to nodes, including remote monitoring and resetting of individual nodes without effecting the uptime of the entire system. Disk cloning is a valuable feature for large cluster systems because it allows software and other updates to be installed on one node and automatically distributed to the entire system. All features are architecture independent."
"Connectivity is provided using ethernet, Myrinet or Dolphin interconnect technology. The company was formed in 1982 by Stephen Fried, physicist and coinventor of the HF chemical laser (Star Wars). Microway has provided state of the art products for high speed numeric processing to the university and government marketplaces for over 18 years. Microway is API's largest US customer for UP2000 and 21264DP Alpha motherboards and processors. The company designs proprietary rack chassis for maximum nodes per cubic foot, and has a fine reputation for delivering fully configured clusters that work! Microway has configured Linux based systems since the early days of the Red Hat releases. Today we specialize in large Beowulf clusters for customers with HTPC applications from chemical engineering, mechanical engineering, CFD, molecular modelling, simulations, biogenetic research to designing America's cup racing keels, and jet engines. Our customers also include ISPs and other ecommerce companies."
For what it is worth, I do remember Microway from years of reading PC Magazine and drooling over their co-processor (i810 and other) boards and fancy matching compilers. Although I have no clue as to where they stand in the grand scheme of cost-beneficial turnkey beowulf systems (so you'll have to visit their website and talk to them and find out for yourself, which you'd do anyway with ALL of these vendors if you had any sense) I will say that in one sense, they were a direct philosophical predecessor of beowulfery. They made an attempt to achieve the same goal of providing commodity supercomputing at an affordable price in a readily available platform. Of course, they tried to achieve it by putting 2-4 very high end processors on a single ISA bus card with very much custom and proprietary high end compilers and a unique architecture, but there is nothing wrong with that, especially in the context in which they were working. For a long time, they probably led the world in cost/benefit measured in minimum commercial cost/FLOP in a single platform. And they are sending me a T shirt, so they must be decent folks.
One of the things Microway sold in the old days was libraries from the Numerical Algorithms Group (NAG). Here is a communication from them:
NAG is a software company that specializes in reusable mathematical and statistical software components. We have been in business for 30 years and have been involved in many projects with the Department of Energy, Department of Defense as well as being able to help many NPACI and Alliance partners. (There are about 40 site licenses within the NPACI and Alliance members which makes collaboration and sharing of software containing NAG very easy.) NAG recently released a new version of the NAG Parallel Library that utilizes MPI for Beowulf computers. The product works with either PGI's compiler or the Gnu product.
All of NAG's products are available for "test drives". NAG will also be coming out with a version of our SMP Library for Intel/Linux next year.Best regards, Tony Nilles
VP Sales and Marketing
NAG - The World Leader in Numerical Software Components
PH: 630-971-2337 x 207
Note that NAG has promised to "try to find something different from the usual t-shirts (since your kids seem to be well stocked!)" Clearly creative and intelligent thinkers!
IDE/SCSI RAID devices (towers of IDE drives on a dedicated RAID controller that attaches to your system via a SCSI interface) are cheap ways to build moderately large (60-240 GB) redundant disk arrays. The chassis and controllers tend to cost perhaps $2000 or thereabouts, and then one just adds OTC IDE disks. With UDMA disks costing less than $4/GB and dropping fast in sizes up to 50-60 GB and rising fast (as of 2/2001) one can fill an enclosure with over 200 GB for less than $1000 more.
In any event, after mentioning their existence on the list, several people asked who makes and where to find these beasts. At least one manufacturer is here and I believe some are resold here. This particular unit got a nice testimonial on the beowulf list -- check the list archives for details
Another suggestion comes from Greg Lindahl of HPTI, above : "Another manufacturer is http://3ware.com/, but again you have to buy through a reseller. 3ware is very Linux friendly and provides a GPLed driver. It *looks* like a SCSI disk in software, but you can do RAID 0, 1, and 1+0. Performance approaches 100 MB/s. But the best part is that the expensive 8-disk unit is only $400 for the card."
However, I'm sure a few minutes with any web search engine would find more and possibly better/cheaper/different ones. A second way to find them, and lots of other good stuff besides, is to talk to some of the turnkey beowulf or general hardware vendors listed above. At least one of them Linux NetworX, wrote me explicitly to offer to help anyone select from "quite an array of storage options" that they carry for their cluster customers.
Warning! Anyone actually contemplating one of these devices is well-advised to read Josip Loncaric's remarks Re: IDE-SCSI RAID units (February 12, 2001) on the beowulf list archives -- RAID1 units like the 3ware solution work fine but CAN take two hours or more to rebuild an absolutely reliable mirror after a crash/reboot.
Ventura Tech assembles "server class memory" with lots of certification and guarantees, pretested for various popular high end motherboards (such as those that might well be selected for a high performance beowulf cluster). Here is their synopsis (from Sam Lewis):
"Many will say that `memory is memory'. While that used to be the case, memory today is anything but standard. With the onslaught of new technologies, it has been a challenge for integrators and OEMs to maintain true compatibility. To that end, Ventura Technology Group (VTG) is a tremendous resource, since we design and manufacture the highest quality server class memory. While our products are compatible among a wide range of platforms (SUN, IBM, HP, APPLE, COMPAQ, DELL, HP, IBM, INTEL, TYAN), our focus will always be on higher density modules (128MB, 256MB, 512MB, 1GB). Rounding out vital service offerings, VTG stretches ahead of the competition in the following areas:
Every module design is manufactured to exceed OEM specifications
* Certifications with major OEM's (Tyan, Intel, Supermicro) insures constant compatibility
* All memory technologies are manufactured in our own ISO9001 certified facility
* All memory modules are manufactured using top grade DRAM and SDRAM (Samsung, Micron, NEC, etc...)
* International and Domestic supplier relationships insure competitive pricing and consistent supply
* Lifetime warranty with the benefit of cross shipment where needed
Along with all of the above benefits, VTG continually provides competitive prices."