Notes on the Austin Meta-Blog
From time to time, we are asked for information on how we set up this site, and to share the tools we used to do so. I thought it would be a good idea to post a note with this information. If you aren't interested in those details, then keep moving ...
First, let's get the bad news out of the way. This web site was implemented with custom code. We wish we could share the tools we've developed with the public, but they just aren't in a condition to do so. Someday we'd like to produce an open source package of what we've done, but it's been stuck on the To Do list for some time.
The primary contributors are Adina Levin (architecture and prototypes), Chip Rosenthal (backend development) and Adam Rice (frontend development).
The web site you see is our third implementation. The phase zero design, where we prototyped the notion of aggregating Austin-related articles, was built at the Internet Topic Exchange. This is a free service and may be useful for people who want to setup a quick-n-easy trackback aggregation site.
This site was setup end of January 2003, and within a few weeks it was obvious that people were interested in an Austin-related meta-blog. Even without significant publicity, we were seeing several postings a week, and bloggers in the community were talking about it.
The next step, phase one, was to build our own meta-blog. We wished to move off the Internet Topic Exchange so that we could have better control over the presentation and features of the site, as well as hosting it under our own domain.
The phase one meta-blog was constructed using the Movable Type weblog package as a backend. A simple CGI script (python2) was used to receive trackback pings, encapsulate them into a weblog entry, and insert them into the blog. (If you are interested in this script, contact me. I can probably dig it up.)
I was concerned about the scalability of this implementation. I didn't feel comfortable the MT backend would hold up if the meta-blog grew to significant size or encountered substantial posting. Also, there was loss of information as the trackback message was being encapsulated into a blog entry. It seemed more appropriate to store the trackbacks directly into a database.
This led us to phase two, the current implementation. We are using a MySQL database backend to store the trackbacks. The backend is coded in perl. The backend includes tools to receive the trackbacks, generate a static version of the current entries, and perform dynamic searches and archive retrievals. The frontend uses php.
The item at the top of my To Do list is to add a blog registry. This would allow us to do things like link blog names to their front-page URL, and provide higher level of privileges for people with registered blogs. The next item after that is producing an open source distribution of the toolset. Unfortunately, there hasn't been a lot of progress on these in recent months.
I think our experiment has been successful, and I think two key things contributed to that success. First, when creating stuff, we've kept things as simple as possible. We deployed the phase one (Movable Type backend) version because it gave us a way to do so quickly and easily. That provided time to go back and work on the more sophisticated version (MySQL backend).
Second, wherever possible, we've tried to eliminate barriers to participation. We provide a manual form for people who cannot do trackbacks. Our system is completely open, which has made it easy for people to join and participate. There have been few incidents of vandalism, and in the few cases when there have been questions about the appropriateness of an entry, some off-line discussion resolved the situation.
The most frustrating and significant limitations are related to the Movable Type software and trackback implementation. The amount of information contained in a trackback message is limited and not extensible. For our purposes, it would be very helpful if we could also get the top-level URL of a blog (only the title is provided) and the publication time of the entry.
Another problem is a Movable Type misfeature. Our members typically setup a blog category (say "Austin") and configure their software to issue an automatic trackback ping whenever an article is placed in this category. If somebody reclassifies an old article into this category then the meta-blog is pinged. Also, if somebody enables the automatic trackback capability on an existing category, all of the existing articles in the category are sent to the meta-blog. If the trackback contained publication date information, we could filter out these spews. But it doesn't, so this has been a recurring problem.
We have been averaging 100-150 visits/day, with the peak occurring during the SXSW Interactive festival. The database currently contains 399 published entries from 49 different weblogs, dating back to January 28 of this year. That's an average of approximately 2.5 entries posted a day.
During this time our community has grown and come closer. We've had a couple of mentions in the local media. I believe these can be attributed, in large part, to the success of our meta-blog.