Reply to thread

Message

<blockquote data-quote="Brackson" data-source="post: 242975" data-attributes="member: 34747">Here's a looping version of this script with proxy support, so you don't get firewalled.[code]#!/usr/bin/env pythonimport urllib2, time, randomproxies = [proxy.strip() for proxy in open('proxies.txt', 'r')] # Setting up the proxies.def store_into_file(proxies):&nbsp; random_proxy = random.choice(proxies)&nbsp; proxy = urllib2.ProxyHandler({'http': random_proxy})&nbsp; try:&nbsp; url = 'http://google.com/' # URL that you want to mine.&nbsp; data = urllib2.urlopen(url).read() # Get the HTML source of URL.&nbsp; current_time = time.strftime('%H:%M:%S', time.localtime()) # Get the current time so we can use if for the txt filename.&nbsp; r = open('%s.txt' % (current_time), 'w') # Create the file.&nbsp; r.write(data) # Put the source in the file.&nbsp; r.close() # Close the file.&nbsp; except:&nbsp; raisewhile True:&nbsp; store_into_file(proxies)[/code]<a href="http://pastie.org/8451768" target="_blank">(with syntax formatting)</a>Include a 'proxies.txt' file with a proxy list separated by line breaks in the same directory, and if you have a good proxy list, you won't be firewalled.</blockquote>

[QUOTE="Brackson, post: 242975, member: 34747"] Here's a looping version of this script with proxy support, so you don't get firewalled. [code] #!/usr/bin/env python import urllib2, time, random proxies = [proxy.strip() for proxy in open('proxies.txt', 'r')] # Setting up the proxies. def store_into_file(proxies): random_proxy = random.choice(proxies) proxy = urllib2.ProxyHandler({'http': random_proxy}) try: url = 'http://google.com/' # URL that you want to mine. data = urllib2.urlopen(url).read() # Get the HTML source of URL. current_time = time.strftime('%H:%M:%S', time.localtime()) # Get the current time so we can use if for the txt filename. r = open('%s.txt' % (current_time), 'w') # Create the file. r.write(data) # Put the source in the file. r.close() # Close the file. except: raise while True: store_into_file(proxies) [/code] [SIZE=1][URL='http://pastie.org/8451768'](with syntax formatting)[/URL][/SIZE] Include a 'proxies.txt' file with a proxy list separated by line breaks in the same directory, and if you have a good proxy list, you won't be firewalled. [/QUOTE]

Verification

Reply to thread

Connect with us

Newest members