If I didn’t know any better, I’d say Lycos and Google were in cahoots. From my webserver logs yesterday:
209.202.221.139 - - [10/Dec/2004:02:41:35 -0800] "HEAD /LenniFan/Discography/Audio/tensongslive/Perhaps%20Perhaps%20Perhaps.mp3 HTTP/1.0" 200 0 "-" "Lycos-News-Xml-Fetcher"
66.249.66.100 - - [10/Dec/2004:02:41:38 -0800] "GET /LenniFan/Discography/Audio/tensongslive/Perhaps%20Perhaps%20Perhaps.mp3 HTTP/1.1" 200 91810 "-" "Mediapartners-Google/2.1"
66.249.66.100 - - [10/Dec/2004:02:41:39 -0800] "GET /LenniFan/Discography/Audio/tensongslive/Perhaps%20Perhaps%20Perhaps.mp3 HTTP/1.1" 200 491649 "-" "Mediapartners-Google/2.1"
61.11.93.167 - - [10/Dec/2004:02:41:52 -0800] "GET /LenniFan/Discography/Audio/tensongslive/Perhaps%20Perhaps%20Perhaps.mp3 HTTP/1.1" 200 445050 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
And a little later:
209.202.221.139 - - [10/Dec/2004:10:00:15 -0800] "HEAD /LenniFan/Discography/Audio/tensongslive/La%20Chanson%20Triste.mp3 HTTP/1.0" 200 0 "-" "Lycos-News-Xml-Fetcher"
66.249.66.100 - - [10/Dec/2004:10:00:23 -0800] "GET /LenniFan/Discography/Audio/tensongslive/La%20Chanson%20Triste.mp3 HTTP/1.1" 200 2568777 "-" "Mediapartners-Google/2.1"
195.68.12.162 - - [10/Dec/2004:10:00:43 -0800] "GET /LenniFan/Discography/Audio/tensongslive/La%20Chanson%20Triste.mp3 HTTP/1.0" 200 623058 "-" "NSPlayer/8.0.0.4487"
Strangest of all is that every time they appear, those paired hits are followed closely by a single request from an unrelated IP address using a different browser. I’m certain that they’re related, because there are no surrounding page hits and those files aren’t something that one would just stumble upon.
This has been going on for at least a week, with each crawler retrieving only MP3 files. If nothing else, Lycos’s so-called “News XML Fetcher” is way out of its league.