Michael Stahnke wrote some excellent slides on developing infrastructure for the Red Hat Summit, partly based on his experience within Fedora Infrastructure.
The gist if the talk is something most sysadmins somehow know, but management rarely likes to admit: infrastructure needs to grow organically with an organization, and there isn’t one solution that fits every org. Of course, that’s sad news for all those big clunky Systems Management applications that large orgs like to buy and then not use, and a serious warning to anybody who thinks they can follow in their footsteps and improve on them.
Fedora Infrastructure, which consistently builds amazing solutions for the Fedora community, is a model of building infrastructure that many organizations would be well-advised to follow. Most of the building blocks for good infrastructure are available as open-source, and tieing them together to custom-tailored solutions for an organization’s specific needs is getting easier and easier. There is general awareness that the road to success is paved with API’s, not shiny UI’s — there’s plenty of tools now that make building custom (web) UI’s easy these days. And with tools like Qpid and QMF the hairier issues with remoting local API’s and talking to a large number of nodes are being reduced from rocket science to car maintenance.
Two things that stand out in Michael’s presentation are the recurring theme of “Don’t leverage the meatcloud” (I think the term was coined by Luke and Andrew from Reductive Labs) — it’s a poignant description of the state of affairs in systems management. The recommendation to “… not use any product with configurations that are not machine parseable and machine writable” goes straight to the heart of the matter - there is still too much configuration data, even when you only look at /etc, that is next to impossible to manipulate programmatically. Through writing tools like Augeas and netcf, I’ve developed a keen interest in the subject; but that merely scratches the surface of what we’d need to make Unix/Linux systems management more easily automatable. So, let’s all write more little tools that can be tied together to big solutions.