Menu
Forums
All threads
Latest threads
New posts
Trending threads
New posts
Search forums
Trending
What's new
New posts
New profile posts
Latest activity
Members
Current visitors
New profile posts
Search profile posts
Upgrades
Log in
Register
What's new
Search
Search
Search titles only
By:
All threads
Latest threads
New posts
Trending threads
New posts
Search forums
Menu
Log in
Register
Navigation
Install the app
Install
More options
Contact us
Close Menu
Forums
Software Development
Programming
[Python] Twitter archiver
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Reply to thread
Message
<blockquote data-quote="griimnak" data-source="post: 450655" data-attributes="member: 35695"><p style="text-align: center"><span style="font-size: 26px"><strong>TwitterArchiver-py</strong></span></p> <p style="text-align: center"></p><p></p><p>I was bored the other night and decided to mess around with the twitter api.</p><p>This script will download profile media, generate a <u>summary.txt</u> and dump tweet history to <u>dump.json</u>.</p><p></p><p><a href="https://github.com/griimnak/TwitterArchiver-py" target="_blank">https://github.com/griimnak/TwitterArchiver-py</a></p><p>[SPOILER=Code]</p><p>[CODE=Python]</p><p>try:</p><p> import twitter</p><p> import requests</p><p>except ImportError:</p><p> exit("ImportError, please run pip install -r requirements.txt")</p><p></p><p>import json</p><p>import os</p><p>import shutil</p><p></p><p>from config import conf</p><p></p><p>try:</p><p> api = twitter.Api(</p><p> # Fill in config fields</p><p> consumer_key=conf["api_key"],</p><p> consumer_secret=conf["api_secret"],</p><p> access_token_key=conf["token_key"],</p><p> access_token_secret=conf["token_secret"],</p><p> )</p><p>except:</p><p> exit("Error with config values.")</p><p></p><p></p><p>def target():</p><p> """ :returns: target json"""</p><p> try:</p><p> target = api.GetUser(screen_name=conf["target"])</p><p> except twitter.error.TwitterError as err:</p><p> exit(str(err)+" (please fill out config.py)")</p><p> except:</p><p> exit("Configuration error")</p><p> return target</p><p></p><p></p><p>def get_image(url, path):</p><p> """ makes request to image and saves as file object """</p><p> r = requests.get(url, stream=True)</p><p> if r.status_code == 200:</p><p> with open(path, 'wb') as f:</p><p> r.raw.decode_content = True</p><p> shutil.copyfileobj(r.raw, f)</p><p> print("[>] get", url, ">>", path)</p><p> f.close()</p><p></p><p></p><p>def get_tweets(api=None, screen_name=None):</p><p> """ get tweet timeline """</p><p> timeline = api.GetUserTimeline(screen_name=screen_name, count=200)</p><p> earliest_tweet = min(timeline, key=lambda x: x.id).id</p><p> print("[+] fetching tweets before:", earliest_tweet)</p><p></p><p> while True:</p><p> tweets = api.GetUserTimeline(screen_name=screen_name, max_id=earliest_tweet, count=200)</p><p> new_earliest = min(tweets, key=lambda x: x.id).id</p><p></p><p> if not tweets or new_earliest == earliest_tweet:</p><p> break</p><p> else:</p><p> earliest_tweet = new_earliest</p><p> print("[+] fetching tweets before:", earliest_tweet)</p><p> timeline += tweets</p><p></p><p> return timeline</p><p></p><p></p><p>def check_dir(dir):</p><p> """ make dir for target if not exists """</p><p> if not os.path.exists(dir):</p><p> print("[+] Creating directory for target..")</p><p> os.makedirs(dir)</p><p></p><p># Work</p><p>target = target()</p><p>work_dir = "data/"+conf["target"]+"/"</p><p>summary = f'''[*] Bio url: {target.url}</p><p>[*] Register date: {target.created_at}</p><p>[*] Tweets: {target.statuses_count} | Followers: {target.followers_count} | Following: {target.friends_count} | Likes: {target.favourites_count}</p><p>[*] Location: {target.location} | Name: {target.name}</p><p>[*] Latest tweet ({target.status.created_at}) </p><p> "{target.status.text}" ({target.status.retweet_count} Retweets)</p><p> source: {target.status.source}</p><p>'''</p><p></p><p># Check dir</p><p>check_dir(work_dir)</p><p></p><p>print("[+] Target:", target.screen_name)</p><p>print("[+] Downloading profile and banner image..")</p><p># Save media</p><p>get_image(target.profile_image_url, work_dir+"profile.jpg")</p><p>get_image(target.profile_banner_url, work_dir+"banner.jpg")</p><p>print("[*] Done.")</p><p></p><p># Generate summary</p><p>with open(work_dir+"summary.txt", "w+", encoding="UTF-8") as f:</p><p> print(summary)</p><p> f.write(summary)</p><p> # friends</p><p> count = 0</p><p> print("[+] Fetching friends..")</p><p> friends = api.GetFriends(screen_name=conf["target"])</p><p> f.write("[+] Fetching friends..")</p><p> for u in friends:</p><p> count+=1</p><p> f.write("\n@"+u.screen_name)</p><p> print("[*] Target has", count, "friends")</p><p> print("[*] Summary generated to", work_dir)</p><p> f.close()</p><p></p><p># Dump</p><p>timeline = get_tweets(api, screen_name=conf["target"])</p><p></p><p>with open(work_dir+"dump.json", "w+") as f:</p><p> print("[+] Dumping full tweet history to", work_dir, "dump.json")</p><p> for tweet in timeline:</p><p> f.write(json.dumps(tweet._json))</p><p> f.write("\n")</p><p> if(conf["print_timeline"] == True):</p><p> print(f'''[+] Tweet ({tweet.created_at})</p><p> "{tweet.text}"</p><p> </p><p> source: {tweet.source}\n''')</p><p> print("[*] Done.")</p><p> f.close()</p><p></p><p>[/CODE]</p><p>[/SPOILER]</p><p></p><p><strong><span style="font-size: 18px">Installation:</span></strong></p><p>[CODE=bash]</p><p>pip install -r requirements.txt</p><p>[/CODE]</p><p></p><p></p><p><strong><span style="font-size: 18px">Usage:</span></strong></p><p>- Configure config.py with your api token. <a href="https://python-twitter.readthedocs.io/en/latest/getting_started.html" target="_blank">(how to get tokens)</a></p><p>[CODE=Python]</p><p>conf = {</p><p> "target": "griimnak",</p><p></p><p> "api_key": "",</p><p> "api_secret": "",</p><p> "token_key": "",</p><p> "token_secret": "",</p><p></p><p> "print_timeline": False</p><p>}</p><p>[/CODE]</p><p></p><p>- Launch</p><p>[CODE=bash]</p><p>python source.py</p><p>[/CODE]</p></blockquote><p></p>
[QUOTE="griimnak, post: 450655, member: 35695"] [CENTER][SIZE=7][B]TwitterArchiver-py[/B][/SIZE] [/CENTER] I was bored the other night and decided to mess around with the twitter api. This script will download profile media, generate a [U]summary.txt[/U] and dump tweet history to [U]dump.json[/U]. [URL]https://github.com/griimnak/TwitterArchiver-py[/URL] [SPOILER=Code] [CODE=Python] try: import twitter import requests except ImportError: exit("ImportError, please run pip install -r requirements.txt") import json import os import shutil from config import conf try: api = twitter.Api( # Fill in config fields consumer_key=conf["api_key"], consumer_secret=conf["api_secret"], access_token_key=conf["token_key"], access_token_secret=conf["token_secret"], ) except: exit("Error with config values.") def target(): """ :returns: target json""" try: target = api.GetUser(screen_name=conf["target"]) except twitter.error.TwitterError as err: exit(str(err)+" (please fill out config.py)") except: exit("Configuration error") return target def get_image(url, path): """ makes request to image and saves as file object """ r = requests.get(url, stream=True) if r.status_code == 200: with open(path, 'wb') as f: r.raw.decode_content = True shutil.copyfileobj(r.raw, f) print("[>] get", url, ">>", path) f.close() def get_tweets(api=None, screen_name=None): """ get tweet timeline """ timeline = api.GetUserTimeline(screen_name=screen_name, count=200) earliest_tweet = min(timeline, key=lambda x: x.id).id print("[+] fetching tweets before:", earliest_tweet) while True: tweets = api.GetUserTimeline(screen_name=screen_name, max_id=earliest_tweet, count=200) new_earliest = min(tweets, key=lambda x: x.id).id if not tweets or new_earliest == earliest_tweet: break else: earliest_tweet = new_earliest print("[+] fetching tweets before:", earliest_tweet) timeline += tweets return timeline def check_dir(dir): """ make dir for target if not exists """ if not os.path.exists(dir): print("[+] Creating directory for target..") os.makedirs(dir) # Work target = target() work_dir = "data/"+conf["target"]+"/" summary = f'''[*] Bio url: {target.url} [*] Register date: {target.created_at} [*] Tweets: {target.statuses_count} | Followers: {target.followers_count} | Following: {target.friends_count} | Likes: {target.favourites_count} [*] Location: {target.location} | Name: {target.name} [*] Latest tweet ({target.status.created_at}) "{target.status.text}" ({target.status.retweet_count} Retweets) source: {target.status.source} ''' # Check dir check_dir(work_dir) print("[+] Target:", target.screen_name) print("[+] Downloading profile and banner image..") # Save media get_image(target.profile_image_url, work_dir+"profile.jpg") get_image(target.profile_banner_url, work_dir+"banner.jpg") print("[*] Done.") # Generate summary with open(work_dir+"summary.txt", "w+", encoding="UTF-8") as f: print(summary) f.write(summary) # friends count = 0 print("[+] Fetching friends..") friends = api.GetFriends(screen_name=conf["target"]) f.write("[+] Fetching friends..") for u in friends: count+=1 f.write("\n@"+u.screen_name) print("[*] Target has", count, "friends") print("[*] Summary generated to", work_dir) f.close() # Dump timeline = get_tweets(api, screen_name=conf["target"]) with open(work_dir+"dump.json", "w+") as f: print("[+] Dumping full tweet history to", work_dir, "dump.json") for tweet in timeline: f.write(json.dumps(tweet._json)) f.write("\n") if(conf["print_timeline"] == True): print(f'''[+] Tweet ({tweet.created_at}) "{tweet.text}" source: {tweet.source}\n''') print("[*] Done.") f.close() [/CODE] [/SPOILER] [B][SIZE=5]Installation:[/SIZE][/B] [CODE=bash] pip install -r requirements.txt [/CODE] [B][SIZE=5]Usage:[/SIZE][/B] - Configure config.py with your api token. [URL='https://python-twitter.readthedocs.io/en/latest/getting_started.html'](how to get tokens)[/URL] [CODE=Python] conf = { "target": "griimnak", "api_key": "", "api_secret": "", "token_key": "", "token_secret": "", "print_timeline": False } [/CODE] - Launch [CODE=bash] python source.py [/CODE] [/QUOTE]
Insert quotes…
Verification
Post reply
Forums
Software Development
Programming
[Python] Twitter archiver
Top