|
|
Beowulf Resources (with abstracts)
This page is devoted to beowulfery and cluster computing. There
in an online book on cluster engineering, talks and tutorials, and
software packages. You can either click directly to a resource page on
the navigation bar above or browse the list below, where each link is
accompanied by a short abstract.
I cherish feedback on any of these resources. To initiate a
discussion, contact me.
- Engineering a Beowulf-Style Compute
Cluster
This is a local copy of my online book on beowulf style
cluster
engineering. It is (and will likely always be) a work in progress, so
check the revision number and dates from time to time to see if new
material has been added.
- wulfware
-
Wulfware: A Cluster Monitoring Suite
by Robert G. Brown (rgb)
wulfware Version 1.0.1
Contents
Back to top
Description
wulfware is a suite of applications that provide remote monitoring
information on a beowulf- or GRID-style compute cluster, a server farm,
or LAN. It consists of the following tools:
- xmlsysd, a daemon that runs on the host to be monitored.
xmlsysd can be run under xinetd or as a standalone forking daemon. It
extracts information from /proc, from systems calls, and from selected
tools or files, packs it into a set of XML tags, and returns that
information to the connected monitor host(s). The daemon can be
throttled to return only the information a monitor interface requires to
support a particular display.
- wulfstat, an ncurses based monitor interface. wulfstat
permits one to monitor an entire cluster or LAN. Statistics it provides
include a vmstat-like display, load averages only, network statistics
only, memory information only, system configuration information and duty
cycle only, and running processes with or without the full command
line. A cluster or LAN can easily be defined with a few simple
commands in an XML-based "wulfhosts" file.
- wulflogger, a command line monitor interface that extracts
more or less the same data as wulfstat in similarly named displays and
writes them in formatted lines to stdout. This data format can easily
be parsed by scripting languages such as perl or python, or it can be
piped into a file for an archival record of cluster activity. The data
can thus be used to generate a variety of reports, including web or
graphical reports.
- wulfweb, an perl-cgi script to generate a
web view of cluster statistics using wulflogger. Currently this is
basically a working perl template for data extraction that can be
modified into whatever kind of display one likes.
- libwulf, the common library upon which wulfstat and
wulflogger are based. It encapsulates the routines that parse wulfhosts
and build the requisite data structures, manage the connections to all
hosts, and poll all connected hosts and extract the desired data from
the xmlsysd return.
The libwulf core can be recycled into a real e.g. GTK GUI version of
wulfstat (and beyond) but so far I haven't had time to mess with this.
Eventually I'll tackle this task. If anyone else wants to write
and contribute back a GUI interface, feel free; I will cheerfully add
your contribution to the wulfware collection.
If you use these tools you may want to join the Wulfware
Mailing List. This is a low traffic list that is used primarily for
announcments (such as the release of a new version) and limited
reporting of bugs or discussion of features. Check out the archives to
get a feel for the level of traffic.
Back to top
DEVELOPER'S NOTES
Starting with 0.6.0, libwulf is
compiled with gcc's -std=c99 option. This is required to access e.g.
unsigned long long ints, which are in turn required by the 2.6.x kernel
which contains a number of parameters in /proc that were formerly
unsigned long ints. Libwulf has also been modified to accomodate
xmlsysd changes associated new fields (and hence new tags) and changes
in format in /proc in the 2.6 kernel as well, although this should not
affect compatibility/usability with 2.4 kernels. However, it may
produce problems on systems with older gcc's. Please report any
problems to the wulfware list or directly to rgb.
- cpu_rate
This is the current snapshot of the cpu_rate
Microtimer harness and microbenchmark program. I use this
program to do quick-and-dirty benchmarks of a variety of systems
performance parameters. The timing harness itself is at least modestly
advanced -- it uses the CPU cycle counter instead of gettimeofday where
possible, for example (on all post-Pentium Intels, post-Athlon AMDs).
There are tests for memory performance, "arithmetic" performance,
transcendental (savage benchmark) performance. In many cases
(specifically those involving vectors and memory) the size of the
vectors can be varied and the effects of cache size and hierarchical
memory explored.
This version of the code supports the relatively simple and
"object-like" addition of code fragments to be timed. There is a
fairly long list of operations and fragments it already times (enter
"cpu_rate -l" to get a current list). It is in beta mode and
may have bugs. There may well be better timers out there (e.g.
lmbench) only perhaps. not so easy to use.
Because of the nature of this program I recommend that you grab the
tarball and work with it rather than simply build/install either the
source rpm or the provided (RH 9) binary rpm. The latter "should work",
but part of the fun of a benchmark/timer is playing with build options
and seeing how it works. There are also some (probably broken in the
current release) scripts in the source directory that in previous
versions scanned e.g. across the -s size option for the vector
benchmarks; these scripts, with a bit of repair, would make it pretty
easy to generate a graphable file of e.g. stream or memory read/write
with and without random shuffling.
I hope that you find cpu_rate useful.
- A Model for Cluster Computing at
Duke
This is a white paper on how Duke might economically set up a
"public" or "shared" cluster compute environment here without offending
the Gods (the granting agencies that might participate) and in a
maximally inexpensive and cost-beneficial manner.
Note Well: This is a Request for Comments only;
please return comments and corrections to rgb for encorporation into what I
hope to be a dynamic document associated with a project, rather than a
static prescription that may, in fact, be heavily flawed.
- An Introduction to the Beowulf
Design
This is a reasonably complete (some would say overcomplete:-)
rendition of a standard introduction to beowulfery, developed from
previous talks on the same subject I've given. As is frequently the
case with talks, the pdf and ps images are likely to be "prettier" than
the online version -- I haven't quite gotten a perfect
latex2html-compatible talk style set up yet.
- Zen and the Art of Beowulf
Clusters
Zen and the Art of Beowulf Clusters
by Robert G. Brown (rgb)
This is a 2006 update of my standard beowulf intro talk, with a Zen
twist. It was last presented to the Science Club of Durham Technical
Institute on 11/7/06, and is being posted so they (and you, if you are
reading this) can access it to look over the content and linked
resources.
Good luck! Feel free to contact me with e.g. bug reports or
problems.
- The Critical Scaling of the Helicity
Modulus of the O(3) classical Heisenberg Ferromagnet
A presentation to the Duke HPC community on my ongoing research into
the critical scaling of the helicity modulus, via Monte Carlo
computations carried out on a cluster.
- Why I code in
C
I can no longer remember where or when I found this while mousing
around on the internet, but it is clearly something anyone seriously
considering the choice between C and C++ should read before investing
their future in one or the other.
- Charles L. Hedrick's Introduction to
the Administration of an Internet Based Local Network (Copyright
1988 Charles L. Hedrick, see link for original copyright notice).
-
Truly a classic, this work (and it's companion, next) were my
original "online guides" in TCP/IP networking (and networking in
general). You'd think they'd be dated, sixteen years later, but of
course TCP/IP itself hasn't changed, ethernet hasn't fundamentally
changed -- amazingly, these are still some of the best simple
introductions available anywhere on or off the net. Thank you, Charles
L. Hedrick, whereever you might be!
- Charles L. Hedrick's Introduction to
the Internet Protocols (Copyright 1987 Charles L. Hedrick, see link
for original copyright notice).
-
The companion work to the local area networking guide above,
this work is just incredible. In addition to beautiful and tediously
typed tty renditions of TCP, IP and ethernet packets showing most of the
relevant details of their headers, this work reviews ports, RFC's, and
more.
|
