body: I work in information security, and one project I have been considering undertaking would involve using a bot (automated web spider) which crawls Tor hidden services for analysis. I am currently in the process of writing the code for the bot. I am concerned that I will run afoul of laws regarding child pornography or copyright infringement, as some hidden services contain such material. The current implementation of the bot does not make a distinction between legal and illegal sites. It does not query me for permission before performing an action (I never give it the "go ahead").
  
  
  Some relevant information:
  
  
  * The bot will be running remotely on a rented server in US jurisdiction.
  * No web resources, legal or otherwise, will be saved to persistent storage.
  * I will not be viewing any material the bot accesses. I will only view its analysis.
  * I cannot keep a site blacklist. The bot will crawl as far as it can.
  * The crawling activity will not interfere with the functionality of the website.
  * Only public resources that can be accessed by anyone will be crawled.
  * The bot may attempt to sign itself up to websites if registration is necessary.
  
  
  I believe that it is likely, if not certain, that the bot will end up accessing illegal material at least once. When it does so, it will fetch the webpage into memory and analyze it. This analysis will be saved to disk. The analysis will contain information regarding anomalies in image file formats, obfuscated JavaScript, non-standard use of CSS, rare HTML tags and uses, etc. It will also save the URL. After analyzing the page, it will continue crawling as a typical web spider, following links.
  
  
  Am I responsible for any laws broken by the bot, given that I am not intending for it to access illegal material but am aware that there is a distinct possibility that it will? Is there anything I should be aware of to reduce the risk of the bot taking an illegal action which I would be responsible for? Given that one could argue (correctly, but out of context) that I am writing a program that signs itself up to illegal websites and downloads illegal material, this makes me very weary.
  
  
  
title: Automated analysis of potentially illegal material using a web spider