Comments on: The Ethics of Web Crawling http://anonymousprof.com/the-ethics-of-web-crawling/ Ramblings and Ravings of an Academic Wed, 19 Nov 2008 04:10:42 +0000 http://wordpress.org/?v=2.5.1 By: Anonymous Prof » Blog Archive » The Legality of Web Crawling http://anonymousprof.com/the-ethics-of-web-crawling/#comment-179 Anonymous Prof » Blog Archive » The Legality of Web Crawling Tue, 25 Mar 2008 14:40:42 +0000 http://anonymousprof.com/the-ethics-of-web-crawling/#comment-179 [...] got a great response to from Aaron to my “ Ethics of Web Crawling” post from yesterday and decided to find out where I stood legally on this issue. Luckily, my [...] [...] got a great response to from Aaron to my “ Ethics of Web Crawling” post from yesterday and decided to find out where I stood legally on this issue. Luckily, my [...]

]]> By: Anonymous Prof http://anonymousprof.com/the-ethics-of-web-crawling/#comment-178 Anonymous Prof Tue, 25 Mar 2008 01:37:21 +0000 http://anonymousprof.com/the-ethics-of-web-crawling/#comment-178 Hey Aaron, That's a great point. I checked for a robots.txt file and there was none. As for their TOS, I'm not sure how to interpret the following: Under a section titled: "Proprietary Rights; Confidentiality" "The Website contains the copyrighted material, trademarks and other proprietary information of Kiva and its licensors. Except for that information that is in the public domain or for which you have been given express written permission, you may not copy, modify, publish, transmit, distribute, perform, display, or sell any such proprietary information." I'm not sure if that covers web crawling or not. Also, b/c my goal for the data is academic and not for profit, would my crawling fall under fair use? Perhaps I should consult a lawyer (luckily, my brother is an expert in this area...looks like he'll be getting a call tomorrow). Thanks! -AP Hey Aaron,

That’s a great point. I checked for a robots.txt file and there was none. As for their TOS, I’m not sure how to interpret the following:

Under a section titled: “Proprietary Rights; Confidentiality”
“The Website contains the copyrighted material, trademarks and other proprietary information of Kiva and its licensors. Except for that information that is in the public domain or for which you have been given express written permission, you may not copy, modify, publish, transmit, distribute, perform, display, or sell any such proprietary information.”

I’m not sure if that covers web crawling or not. Also, b/c my goal for the data is academic and not for profit, would my crawling fall under fair use? Perhaps I should consult a lawyer (luckily, my brother is an expert in this area…looks like he’ll be getting a call tomorrow).

Thanks!
-AP

]]> By: Aaron Schiff http://anonymousprof.com/the-ethics-of-web-crawling/#comment-177 Aaron Schiff Mon, 24 Mar 2008 23:23:37 +0000 http://anonymousprof.com/the-ethics-of-web-crawling/#comment-177 Many commercial websites have terms of use that prohibit web crawling to collect data. For example airline sites often will not let you collect their price data in bulk. You should check if the site you crawled has such a policy. Also check the server's robots.txt file to see if they've prevented indexing by search engines. If neither of these things apply, I'd say it's safe to assume that the site does not mind their data being crawled (it will have been crawled by Google etc already anyway). Many commercial websites have terms of use that prohibit web crawling to collect data. For example airline sites often will not let you collect their price data in bulk. You should check if the site you crawled has such a policy. Also check the server’s robots.txt file to see if they’ve prevented indexing by search engines. If neither of these things apply, I’d say it’s safe to assume that the site does not mind their data being crawled (it will have been crawled by Google etc already anyway).

]]>