Dave Winer Illustrates Problem of Referer Log Spamming
Over on Scripting News, Dave Winer addressed the issue of referer log spamming, a technique that pornographic web sites have started using to advertise themselves. Referer logs are automatically generated reports of where visitors to web sites are coming from. When referer logs are not abused, they are an indication of where links to a given web site can be found.
Many weblogs now publicize their referer logs, mainly because they can. It helps give third parties an idea of the so-called flow that a web site gets. However, referer logs can be abused because the browser being used volunteers the referer data to the web server, and this is something that can be forged. This is how the pornography sites are manipulating the referer log on scripting.com.
Winer says, "A couple of weeks ago we finally figured out why porn sites add themselves to referer pages on high page-rank sites: to improve their placement in search engines. Last night at dinner Andrew Grumet came up with the solution. In robots.txt specifically tell Googlebot and its relatives to not index the Referers page. Then the spammers won't get the page-rank they seek."
For an illustration of the sort of referer log spam Winer is talking about, check out the middle third of the Scripting News referer's page. But, don't go there if you are offended by slang terms for human genitalia.
Porn sites operators may be evil, but they often leverage web technology in the most subtle and sophisticated ways. They really have analyzed popular web services, such as search engine spiders and SMTP, and figured out how to use them to marginally increase traffic to their sites.