Well, as I started out my research on my master’s degree, one of my first tasks was to get the abstract and citation information about thousands of scientific articles. I intend to use this data in the machine learning classes.
There are a lot of systems that organize articles. One of them is the ACM Portal. There you can perform really good searching. It’s a very positive point for the system. What they don’t seem to have is an API for automated searching.
Then, I had to write a script (in my favorite language, PHP) to help me out with my task. My first attempt failed badly. After something like 30 requests, the site started giving the “403 Access Forbidden” message.
So, how would I get the information on the 5000 articles that I want? The answer is simple. Proxy. They can detect several requests from the same host but, apparently, they can’t recognize when I distribute those requests over a few proxies.
First, I had to find my list of proxies. Sites like this http://www.proxyvadi.net/21-03-2011-proxy-list-proxy-listesi-7.html offer me just what I wanted. A GOOD list.
What the script does is simple: get a random proxy from the list and use it on the file_get_contents PHP function.
I really hate when I go to some blog/site/whatever and the guy brags about his findings but doesn’t show any results. So, attached to this post you can find the code I wrote to perform my crawling.
Just one tip: handling threads in PHP is not something very simple/pleasant to do. Instead of parallelizing the script, I just run it N times.
Please, write comments :)