The Robot Trap

Testing to see what bots actually visit a basic page on a small website

I’m testing for robots.

I thought it might be interesting to find out how many robots patrol through a regular, low-traffic page like this scavenging for email addresses and other things to steal or exploit.

There are all kinds of things crawling around on the net.

Anytime a page is visited, it is possible to grab the user agent, a piece of text that describes who is coming to read. For you this is your browser (like Firefox) and along with your operating system (Windows 7). This is why pages look different on your iPhone than on your laptop – they can tell what you’re using, and give you something tailored to your device.

But some visitors aren’t humans at all. They’re bots (robots).

Some of them are nice. I’ve even seen some whose policies state geekily that they follow the Three Laws of Robotics.

An example of a harmless bot is Google’s crawler, which meanders about looking for changes so Google can keep its searches up to date.

And then there are the bad guys who look for people who have put their email addresses online. They grab those up in the interest of SPAMing. Others look for blogs where they can leave hundreds of comments about the latest sex enhancement drugs.

Well, I was just wondering what’d I’d find if I kept track of all everyone that visited a humble, little page on a humble, little site… and if the results are cool, I’ll post them.

In case you’re curious, your user agent is:

$agent = mysql_real_escape_string( $_SERVER['HTTP_USER_AGENT'] );
$sql = "INSERT INTO jg_user_agents VALUES( null, '$agent', NOW() )";

echo ' >> ‘ . $agent . ‘‘;

mysql_query( $sql );


me know
what you think
just remember i have feelings too