Arts and Webwatcher for Document Management
Originally published in SysAdmin, Jul 2006, Vol 15, No 5
Introduction
The creation and maintenance of documentation arguably constitutes the most important job we have as systems administrators. It informs every aspect of our work in ways so pervasive and deep that we hardly consider them documentation any more. We comment configuration files. We send email to users, explaining how to do things. We stand around the water cooler, talking about past coworkers and debating best practices. We write more email, for other sysadmins or for ourselves, hashing things out further. All this documentation has tangible value. The users can figure out how to get their work done. The programmers know why you set things up that way, and why you'll deny their request to change it. You can remember six months from now why you chose not to use the CVS pserver, and explain it to your boss. Good documentation can save you work, but more importantly, it can inform your planning, giving you a solid basis for future work.
Why, then, does writing documentation so often get the short end of the stick? In my conversations with other sysadmins, it seems like we always agree that writing documentation ranks on the desirable job duties scale somewhere below taking out the trash and barely above tidying up our cubicles. Or else, we think we might want to do it someday, but not until we have more time. It gets put off again and again. Then somebody absolutely must have information about the Frobnicator 3000, and we write just enough to help them get by. And naturally, the documentation produced this way goes out of date almost as quickly as it gets written, and we subsequently forget about it or ignore it. Maintaining old documentation seems even less popular than generating new - a source of annoyance exacerbated by our users' disturbing tendency to consider old documentation accurate.
If your work habits are like mine, you probably write documentation in a completely ad-hoc fashion, satisfying individual requests as they come. You probably get it where it needs to go via email, too. And it probably won't surprise you to hear that I've rewritten some end-user instructions at least a half dozen times, with only minor variations between them. Sometimes it happens that I notice that I've written something before, and I grep my sent-mail folder for it, clean it up, and send it out again. Only rarely does any particular piece of documentation get requested often enough for me to make a web page out of it. Naturally, those pages go out of date quickly.
Never mind historical context. Requests-for-quote, historical digressions, meeting summaries, commentaries on the local culture, impassioned essays about password hygiene. All get produced as one-offs and sent to the appropriate mailing lists with hardly a second thought. There they moulder in the mountainous archives, never read again. They don't have even the slightest chance of making it into out-of-date webpages.
All of that changed for me when I heard about a pair of simple tools: 'Arts'[1] and 'Webwatcher'[2]. They changed the way I handle all manner of administrative documentation. Each does one thing, and does it well. Neither alone can change your life, but together they provide just enough functionality to begin to make sense of the madness.
Arts
Chris Dent wrote 'Arts' long ago while working at a small Midwestern ISP, to address just these sorts of documentation issues. It consists of a pair of scripts which provide a document repository and manage metadata on that repository. The first of these, 'arts.gw', gets configured as a delivery pipeline in your MTA or procmail. It takes incoming messages, adds some metadata (including a default expiration time), and sticks them in a predefined location as web documents. The second, 'arts.ct', gets called from cron, or more rarely the command line, and generates a web page with a table of contents of all of the Arts documents, organized by section.
Arts has a straightforward configuration and installation process, and I refer you to its enclosed documentation. I will note, however, that Arts enables anyone who can send messages to it to put data on your disk. Obviously this can cause problems, and you should take steps to restrict who may send messages to Arts and probably also how much disk Arts can use. Some simple notions of restriction come with Arts. You can choose which user groups or address domains can add documents to the repository, for example. However, spoofing email addresses might easily overcome these checks. The dilligent sysadmin will want to restrict things more. Careful configuration of the MTA can help.
Configuring and installing Arts comprises only a small part of its deployment, of course. Developing the habits to use it requires more effort. The process does come easily, but it can take a while before it comes naturally. A typical usage scenario might go like this:
Bob wants to change his password on the locally-situated Frobnicator software. He sends email to the archived systems administrator's mailing list, systems@example-company.com. Alice gets the message via the list and writes a set of instructions detailing password changes on Frobnicator and several other quirky systems. She does a group reply, so that the message goes to both Bob and the systems list, and hits send.
The next day, Carl comes back from vacation and notices the instructions. He remembers that Drew just asked him about Frobnicator passwords the week before, and so he uses his email client's bounce command to send a copy of Alice's instructions to the Arts address. Now, Alice's instructions reside in a central, web-accessible location, and Bob, Drew, and everyone else can benefit from them.
Later, Frances emails systems and asks for help setting her user preferences in Frobnicator. Once again, Alice promptly sends a detailed email to both Frances and the systems list. A conversation ensues between Alice and Carl, and they decide that the message should get preserved. Alice bounces the message to Arts. When Bob needs to update his user preferences, he finds the instructions already written and so doesn't bother to email systems.
I hope you can see by now how even a simple tool like this can help significantly with your documentation tasks, by giving you a filing system that fits (almost) transparently into your workflow. It gives you an organization system too, though. The title of Arts documents get set to the subject line of your email (adjustable later, of course). You can prepend a word or phrase followed by a hash mark to indicate a 'section' in the table of contents that a given document will go into. For example:
To: myarts@example-company.com
Subject: Frobnicator 3000#How to set your password
You go to http://www.example-company.com/frobnicator3000 and click on 'Change Password'. Then you change it. Have a nice day.
If Alice had prepended her emails to Bob and Frances with the Frobnicator section like this example, then the next time arts.ct generated the table of contents her documents would collect together in one place.
Webwatcher
Dent wrote 'Webwatcher' at approximately the same as Arts, but didn't initially anticipate their use together. Webwatcher compares an 'expires on' stamp (added by hand or by arts.gw) in some set of web pages[3] with the file's age on disk. When you run Webwatcher from cron, it generates output when static web content has expired. Hopefully, these complaints will help spur you to keep the files up to date. You can configure Webwatcher to watch multiple different collections of files and complain to the owner recorded in each file's metadata section when it expires. It even has a boss feature, so if you let your work slide for too long, retribution will come down from above.
Webwatcher by itself solves a pretty annoying problem. The real value to the working systems administrator, however, comes from combining Webwatcher and Arts. Now, in addition to your (almost) transparent system for documentation creation and filing, you get periodic reminders (user adjustable, naturally) to keep it all up to date. If you expect some particular document not to change, set it to never expire (live meeting minutes or requests-for-quote, for example), or if another changes all the time, set it expire in just a few days (a running log of progress on some project).
Like Arts, the proper use of Webwatcher requires the internalization of some new practices. In particular, if you want timely documentation, you must consider Webwatcher's nag emails as high-priority tasks. Reviewing and updating your documents shouldn't get put off any longer than necessary. If a particular document sends too much email, then review it anyway, and adjust the expiration period so that it will stay fresh longer. Refrain from simply using 'touch' on documents without review to get Webwatcher to stop complaining. When you do so, you admit that you don't care about your documentation. Why use Webwatcher at all, if you don't care about documentation?
Ultimately, good documentation comes from you. You will have to make the production and maintenance of documentation into a daily habit, like reading email or getting that first cup of coffee in the morning. The discipline to consistently send good email into the document repository or to update old documents must come from you. Once acquired, however, these habits can save you time and work. Tools like Arts and Webwatcher can help ease the process, making the habits easier to acquire and live with, and helping you get to cooler work more quickly.
Footnotes / References
- Arts hails from: http://www.jrbl.org/projects/arts/index.html
- Webwatcher hails from: http://www.jrbl.org/projects/webwatcher/index.html
- Of course, you may change the regular expressions that Webwatcher uses to parse documents, and use it to watch things other than web pages. Discussion of such matters takes us further afield than I wish to go, however.
- Chris Dent, the author of Webwatcher and Arts, still works on projects to make communication and documentation easier at http://www.burningchrome.com/~cdent
Site contents © Joe Blaylock, 2000-2007. All Rights Reserved.
gpg 0b041c44: 8fac d910 6a85 68b8 dc03 0e3D 8bc6 774a 0b04 1c44