Another pet peeve

To Yahoo!, Google, and all of the other web crawlers out there: http://www.example.org/foo and http://www.example.org/foo/ are not the same URI! If you get a 301 Moved Permanently response from your HTTP request, the target is the URI to save in your database and to check in the future.

68.142.249.111 - - [11/Sep/2005:19:23:23 -0400] "GET /pix HTTP/1.0" 301 321 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
68.142.249.111 - - [07/Sep/2005:19:01:43 -0400] "GET /pix HTTP/1.0" 301 321 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
68.142.249.111 - - [29/Aug/2005:22:01:45 -0400] "GET /pix HTTP/1.0" 301 321 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
68.142.249.111 - - [22/Aug/2005:03:42:29 -0400] "GET /pix HTTP/1.0" 301 321 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
68.142.249.111 - - [11/Aug/2005:06:46:15 -0400] "GET /pix HTTP/1.0" 301 321 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
68.142.249.111 - - [03/Aug/2005:18:45:41 -0400] "GET /pix HTTP/1.0" 301 321 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
68.142.249.111 - - [27/Jul/2005:02:45:05 -0400] "GET /pix HTTP/1.0" 301 321 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
68.142.249.111 - - [19/Jul/2005:09:40:07 -0400] "GET /pix HTTP/1.0" 301 321 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
68.142.249.111 - - [11/Jul/2005:14:02:58 -0400] "GET /pix HTTP/1.0" 301 321 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

66.249.65.234 - - [17/Sep/2005:01:39:47 -0400] "GET /pix HTTP/1.1" 301 321 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.172 - - [04/Sep/2005:04:43:58 -0400] "GET /pix HTTP/1.1" 301 321 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Published by

Peter