sitting here tailing logs from apache...

1000% success rate, if you are a bot, you don't ask for favicon.ico

Can we use favicon "port knocking?"

If you ask for / and dont ask for favicon, then yhou are a bot and passed over fail2ban

I need to make this a thing.

@kajer you can do this with mod_rewrite in apache, or otherwise its equivalent in nginx. is favicon the first thing that a regular browser asks for? or even in the top 5?

@Viss

right, i'm thinking of something like:

if you request >20 things w/o one of them being favicon... straight to jail.

@kajer this would make a great fail2ban rule

@kajer @Viss I wonder if you could make use of a link element to point favicon requests to a special endpoint. Something like:


<link rel="icon" href="/i_am_not_a_robot_heres_my_favicon_request.png">

@logan @Viss

yes and no?

When you point to favicon in the < head > section, it becomes a link to parse...

Most browsers when going to a URI, will request favicon all on their own.

@kajer @Viss Hmm, that's a good point too. It seems like you have a pattern identified anyway, I think I was just chasing a hypothetical "what if some bots also request the favicon?" with that train of thought.

@logan @kajer most bots dont fetch .css or .js either. or robots.txt. spotting bots in web logs is super easy.

@Viss @kajer Yup, fair enough. The bots I was seeing in my logs were just requesting HTML documents. I then set up Anubis on the targeted service and kind of stopped caring. I should take a fresh look at that, maybe things changed, but sounds like not in the grand scheme of things.

@Viss @kajer Anything to make visits have less friction for legitimate visitors as well. This seems like this is a nice way to skip the Anubis challenge page and just react based on heuristics.

@logan @kajer its pretty open ended at this point. ctail -f your web logs and watch the background radiation. then hit your site with a real browser. the differences in the logs are night and day. you can also block entirely by user agent too.

@Viss @kajer for someone who uses lynx, this would make a terrible fail2ban rule.

@LinuxAndYarn @kajer wow you use lynx often enough this sort of setup would be a problem?

@Viss @kajer I do, a few times a week, because I use it on some servers to avoid the paywall. I also want to make sure that sites on our Network still produce good text only output for people who use a simpler browser for accessibility reasons.

I suppose a rule that made sure to exclude text-based browsers by ID would be the handy.

@LinuxAndYarn @kajer most bots are easy to spot, so this sort of exclusion is super easy

@Viss @LinuxAndYarn @kajer I use w3m every day, which would have the same problem. Mind you, I don't think many bots are faking their User-Agent as lynx - or w3m.