Warning: Declaration of ThemeSwitcherWidget::update($new_instance) should be compatible with WP_Widget::update($new_instance, $old_instance) in /home/kingston/hitormiss/wp-content/plugins/theme-switcher/theme-switcher.php on line 0

Hit or Miss

2 Jul 2000

Remember how I said I don’t like it when search indexing bots ignore my robots.txt file and spider through my site?

The particular bot I’m mad at is coming from 213.219.19.148 and shows up in my referer logs as the user agent “Katriona.”

I feel like I’m undergoing a denial of service attack from her. For the past 4 days, this bot has continued to spider through my site. Because my site is dynamically driven and has links repeated on every page, the bot keeps following links around and around and arround. Now it seems to be making up links – for example she’s trying to do searches for files like:

“/archives/2000/01/go/http://www.sierra.com/search.”

I don’t know how to stop her. Can anyone please offer me some advice for stopping spiders if I’m running Apache 1.3.12 on Linux?

2 responses so far (Respond)

Gravatar

Okay, so I tried to do a RewriteCond/Rule in my .htaccess file to block the IP number of the indexing bot. For some reason, everytime I tried to add it, it caused MySQL to stop running which knocks my site down.

Matt | 3 Jul 2000
Gravatar

That spider is doing the same to my site.

– Elan

Elan | 6 Jul 2000