Month: August 2003

I recently discovered a subtlety of the /robots.txt file. (For those who don’t know, /robots.txt is a configuration file for web spiders that tells them what URLs not to retrieve from a website.) The issue is this: parsers will not fall back to a more general User-Agent setting if they’ve already matched a specific one. This is actually spelled out in A Standard for Robot Exclusion — If the [User-Agent] value is ‘*’, the record describes the default access policy for any robot that has not matched any of the other records — but I’d forgotten it when I updated my /robots.txt recently to exclude certain additional URIs (removed from my site due to abuse) that Google seems to have such a fondness for that it won’t drop them from its index.

Here’s an excerpt from /robots.txt that shows the error:

User-agent: Googlebot
Disallow: /CBP/

User-agent: *
Disallow: /cgi-bin/
Disallow: /personal/

Imagine my surprise when Googlebot started spidering /cgi-bin/show?user=Webmaster, my address-obfuscating contact form, and pages under /personal/ which poison the lists of e-mail addresses and URIs that address harvesters collect.

I’ve updated /robots.txt to include both sets of URIs for the Googlebot, but (unlike Inktomi’s Slurp) Google doesn’t retrieve /robots.txt after it’s started spidering a site. I may have to extend my .htaccess blocking for the current dance to prevent the big G from grabbing a bunch of (more) useless garbage from my site.

Moral of the story: even if you’ve read the documentation, read it again before making assumptions. As my friend John is fond of saying, when you assume you make an ass of u and Dave.

I’ve just finished de-cruftifying Petroglyphs, inspired by Mark Pilgrim and based on the work of Már Örlyggson. To my view at least, all my URLs are cool and the weblog as a whole is non-application-specific. As with all changes of this sort, please let me know if anything appears broken.

Back at the turn of the twenty-first century there was a huge electrical blackout in North America. Some 50 million people in two countries lost power for an entire day, many for even longer.

I was one of those 50 million people. This is what it looked like from here.

My right wrist has been hurting on and off for the last week, and fairly constantly over the last couple of days. I’ve got reasonably ergonomic setups at home and at work, but I think it’s time I start looking more closely at RSI information.

I’ve been trying to figure out the best way to mark up lyrics for the Sirens website (and, tangentially, my Lenni Jabour fan site). For my purposes, best means having the most flexibility for entry and layout while retaining as much semantic value as possible. There are a few options:

<pre>

Preformatted text is probably the simplest way to go, but it’s also semantically poor. It also suffers from an apparent bug in IE (who’d’a thunk?) which doesn’t allow the font of a <pre> element to be styled.

<p/>/<div/>

Piggin.Net suggests using a separate paragraph for each line. This makes layout much easier, but drops the semantic idea of paragraphs being groups of sentences. On the other hand, the stanzas example does group lines into verses using <div/> elements.

<p/>/<br/>

Paragraphs and line breaks are reasonable and commonly used, though they provide somewhat less-flexible layout options than simple <p/> elements. Classes can be applied directly to each paragraph, however, which allows things like <p class=”chorus”>…</p>.

<p/> plus white-space: pre

Simple paragraphs are probably the most semantically correct, and applying the CSS white-space: pre property allows them to be laid out in lines. Non-CSS2 browsers will see long lines of text for each verse, though, which isn’t the desired effect.

All <div/>, all the time

<div/> throws out all the semantics and most of the style inherent in HTML entirely, leaving everything to CSS. Non-CSS browsers will have a fit, and this isn’t much different otherwise from <p/>/<div/> or <p/>/white-space: pre.

XML markup

Definitely the most semantic, but also the least usable for general browsing. TEI‘s Base Tag Set for Verse seems to be the ultimate method; all of the other XML music formats (like 4ML) are note-based, so it’s impossible to do just lyrics. Layout-wise, if more browsers supported styling inline XML it might be worthwhile, but ultimately there’s nothing to see here. Move along.

Markupbation aside, I’m thinking <p/>/<br/> gives the biggest bang for the buck. Add some classes (verse and chorus) and some CSS (.chorus:before { content: ‘Chorus:’; }) and I’m ready to go.

As Dave Shea says, This is an article I needed to find myself…. Feel free to link gratuitously with [appropriate search term phrases]… so that others may benefit from it.