Outlining the projects I've worked onincluding web security, Bash scripting, and general documentation
Web SecurityI've done plenty of preliminary research into the web security model we're using for customer servers. I've documented our approach as well as a few others and suggested a few minor improvements, some of which have been implemented. I recommend the use of upstream Debian's PHP and the suhosin patch and module along with our current permissions and ownership schema and use of mod mod_security. Of course Apache2 has many more features and capabilities for security than Apache 1.3. As with everything else for maximum safety the whole web stack (apache/php/mysql or whatever and dependent libraries) will need to be regularly updated to bring in new fixes and maintain safety.
Incident ResponseAlong with this I've started to formalize our incident response for web intrusions. With a standard for record keeping we could gain valuable information about the attacks and attacks that trouble our customers and allocate our resources to better defend them as well as share tools and techniques better.
To that end along with the documentation I'm working on I wrote a Web Security quiz to help evaluate sysadmin knowledge of the tools and technologies. It's mostly Unix oriented with just a few Windows questions.
New tricks?One easy win that can save us lots of time and customer annoyance is to to quarrantine and scrub the customer's content during it's migration to remove or disable obvious backdoors and viruses. This would eliminate plenty of nastiness that they currently bring in with them from their previous hosts and save time and trouble later with some minimal upfront work which we can fully automate.
I've given some thought to offering Enhanced Web site security features as an option to some customers and haven't come up with a way that benefits both us and them. Planned features could be suexec,suphp (and possibly even chroots) with only one website to a uid. Another thing that would benefit some of our customers at some trouble to us is per-user tmp folders, since many preconfigured web attack scripts take advantage of the lax permissions of /tmp. There are probably some performance and safety gains to be found in the persistent cgi daemons (like fcgid) but this requires further investigation. Similarly the security implications of Memcached, EAccelerator and the like are unknown.
I'd really like us to have test installs of the web applications our customers use the most on an internal server that we can use as example and testbed. As the requirements for requested web apps get more complex and complicated it will greatly benefit us in safety and efficiency to know what it will take to get the newest version of eg ClipShare, ComusThumbs, ArrowTrader and WordPress running on our install. And of course anything that needs configuration or compilation should be prebuilt and waiting on our Debian repository as Majied has already done for ffmpeg and mplayer among others.
ScriptingI've improved as a scripter in Bash and Ruby over the last year or so entirely from practice with plenty of short scripts and a few long ones along with many hours on the command line and in vim. Among others I've written a short script to clear the mess out of directories that are too full to wildcard and I have a rudimentary Wordpress updater working that needs larger scale testing. Here's an overblown log http://adric.net/hacks/truncat truncator in Ruby. Although a trivial program it demonstrates my coding style pretty well with plentiful documentation and some input-checking.
The vast majority of the utility sysadmins get from the shell is in interactive usage (shell utilities in pipelines, wildcards, simple loops, simple regexes) rather than in writing programs, so I try and document useful command line hacks and am working on a shell usage quiz. I still struggle with all but the simplest Apache mod_rewrite rules and I'm documenting as I do nail down working examples.
CustomersWhenever I have gathered enough information to I have documented customer systems for general understanding and help in troubleshooting. Here are a couple good examples of this work.
ToolsI have documented upstream tools for internal use (SVNServe,SELinux), local tools (Watchdogd,Backdoor_scan), and even the difference between upstream versions and our internal usage as well as general notes such the gotchas for upgrading Sarge machines to stable or how to get Mod_gzip working on our installs. Of course I have lots of pages devoted to research and projects in progress.
WikiFrom the notes Michael Roderick and I took in our first few weeks we started to put together comprehensive new sysadmin documentation. From our notes and his idea I compiled the HowTo page which keeps updated answers to numerous quick questions about tools, procedure, common issues, and where things are kept. I needed a map of the datacenter to explain where some things were, so I made one. I've recently updated http://nat17/dc4-2008.png it with the new fixtures, colors, and cleaner graphics.
I have started using MediaWiki features to help with documentation. I have set up several Categories that I use to track the technologies customers use such as :Category:LoadBalanced and a couple Templates to ease linking to tickets and alerts. I have some ideas about using MediaWiki transclusion for a partially dynamic front page (or a front page) of the wiki that need more testing.