« Hitting the Reset Button | Main | Some Bloggers Suspect that Google Will Segregate Their Sites »

Dave Winer Illustrates Problem of Referer Log Spamming

Over on Scripting News, Dave Winer addressed the issue of referer log spamming, a technique that pornographic web sites have started using to advertise themselves. Referer logs are automatically generated reports of where visitors to web sites are coming from. When referer logs are not abused, they are an indication of where links to a given web site can be found.

Many weblogs now publicize their referer logs, mainly because they can. It helps give third parties an idea of the so-called flow that a web site gets. However, referer logs can be abused because the browser being used volunteers the referer data to the web server, and this is something that can be forged. This is how the pornography sites are manipulating the referer log on scripting.com.

Winer says, "A couple of weeks ago we finally figured out why porn sites add themselves to referer pages on high page-rank sites: to improve their placement in search engines. Last night at dinner Andrew Grumet came up with the solution. In robots.txt specifically tell Googlebot and its relatives to not index the Referers page. Then the spammers won't get the page-rank they seek."

For an illustration of the sort of referer log spam Winer is talking about, check out the middle third of the Scripting News referer's page. But, don't go there if you are offended by slang terms for human genitalia.

Porn sites operators may be evil, but they often leverage web technology in the most subtle and sophisticated ways. They really have analyzed popular web services, such as search engine spiders and SMTP, and figured out how to use them to marginally increase traffic to their sites.

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)


CTDATA Venutures (CTDATA) develops Internet and Intranet applications for corporations and non profit organizations. Our services include:

  • Consulting services for Movable Type and TypePad-based publishing systems (visit our Weblog Improvement website for more information),
  • Financial services business process consulting,
  • Content management system and knowledge management system consulting,
  • Apache web server engineering and hosting,
  • MySQL, Sybase, and Microsoft SQL Server architecture and development,
  • SOAP, REST, and XML-RPC system architecture and programming, including Amazon Web Services and
  • Weblog publishing.
For more information, contact Dave Aiello by email at dave [at] daveaiello.com or call him at +1-267-352-4420.
Copyright © 1995-2010, CTDATA Ventures. All Rights Reserved.
Powered by
Movable Type 4.25