Coder Social home page Coder Social logo

achyuthjoism / tweeds Goto Github PK

View Code? Open in Web Editor NEW
176.0 6.0 33.0 280 KB

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a Tweets and more while evading most API limitations.

Home Page: https://dev.to/achyuthjoism/how-to-scrape-tweets-from-twitter-with-python-using-tweeds-4k2e

License: MIT License

Python 100.00%
hacking-tool osint osint-python twitter twitter-osint python scraper tweets twitter-api

tweeds's Introduction

Due to some changes in Twitter API interface we can't use this tool anymore!!

Tweeds - Twitter Scraping Tool

Scrape tweets from any Twitter user profile. Twitter API alternative to scrape Twitter hashtags, threads, images, videos, statistics, and Twitter history. Export data in JSON and CSV format. This project enables you to extract large amounts of data from Twitter. It lets you do much more than the Twitter API, because it doesn't have rate limits and you don't even need to have a Twitter account, a registered app, or Twitter API key.

Do not forget to star this project.๐ŸŒŸ ๐Ÿ˜

๐Ÿ’ก Features

  • No API Key required
  • No Limit
  • No Authentication required
  • Get tweets from specific user
  • JSON and CSV export for further Data Analysis
  • And the best part it is OpenSource ๐Ÿ˜‰

โœ”๏ธ Requirements

  • Python >= 3.9

โš™ Installation

pip install tweeds

๐Ÿ’ƒ Usage

achyuthjoism-MBP ~ % tweeds

        โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•—โ–‘โ–ˆโ–ˆโ•—โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–ˆโ–ˆโ•—โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•—โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•—โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•—โ–‘โ–‘โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•—
        โ•šโ•โ•โ–ˆโ–ˆโ•”โ•โ•โ•โ–‘โ–ˆโ–ˆโ•‘โ–‘โ–‘โ–ˆโ–ˆโ•—โ–‘โ–‘โ–ˆโ–ˆโ•‘โ–ˆโ–ˆโ•”โ•โ•โ•โ•โ•โ–ˆโ–ˆโ•”โ•โ•โ•โ•โ•โ–ˆโ–ˆโ•”โ•โ•โ–ˆโ–ˆโ•—โ–ˆโ–ˆโ•”โ•โ•โ•โ•โ•
        โ–‘โ–‘โ–‘โ–ˆโ–ˆโ•‘โ–‘โ–‘โ–‘โ–‘โ•šโ–ˆโ–ˆโ•—โ–ˆโ–ˆโ–ˆโ–ˆโ•—โ–ˆโ–ˆโ•”โ•โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•—โ–‘โ–‘โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•—โ–‘โ–‘โ–ˆโ–ˆโ•‘โ–‘โ–‘โ–ˆโ–ˆโ•‘โ•šโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•—โ–‘
        โ–‘โ–‘โ–‘โ–ˆโ–ˆโ•‘โ–‘โ–‘โ–‘โ–‘โ–‘โ–ˆโ–ˆโ–ˆโ–ˆโ•”โ•โ–ˆโ–ˆโ–ˆโ–ˆโ•‘โ–‘โ–ˆโ–ˆโ•”โ•โ•โ•โ–‘โ–‘โ–ˆโ–ˆโ•”โ•โ•โ•โ–‘โ–‘โ–ˆโ–ˆโ•‘โ–‘โ–‘โ–ˆโ–ˆโ•‘โ–‘โ•šโ•โ•โ•โ–ˆโ–ˆโ•—
        โ–‘โ–‘โ–‘โ–ˆโ–ˆโ•‘โ–‘โ–‘โ–‘โ–‘โ–‘โ•šโ–ˆโ–ˆโ•”โ•โ–‘โ•šโ–ˆโ–ˆโ•”โ•โ–‘โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•—โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•—โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•”โ•โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•”โ•
        โ–‘โ–‘โ–‘โ•šโ•โ•โ–‘โ–‘โ–‘โ–‘โ–‘โ–‘โ•šโ•โ•โ–‘โ–‘โ–‘โ•šโ•โ•โ–‘โ–‘โ•šโ•โ•โ•โ•โ•โ•โ•โ•šโ•โ•โ•โ•โ•โ•โ•โ•šโ•โ•โ•โ•โ•โ•โ–‘โ•šโ•โ•โ•โ•โ•โ•โ–‘

        by Achyuth Jois M

usage: tweeds [-h] [-u U] [-s S] [--since SINCE] [--until UNTIL] [--limit LIMIT] [--near NEAR] [--geocode GEOCODE]
              [--year YEAR] [--today] [--verified] [--link LINK] [--videos] [--images] [--media] [--minLikes MINLIKES]
              [--minRetweets MINRETWEETS] [--minReplies MINREPLIES] [--json JSON] [--csv CSV] [--silent]

Scrape twitter user tweets

options:
    -h, --help            show this help message and exit
    -u U                  Users Tweets you want to scrape.
    -s S                  Search for Tweets containing this word or phrase.
    --since SINCE         Filter Tweets sent since date (Example: "2017-12-27 20:30:15" or 2017-12-27).
    --until UNTIL         Filter Tweets sent until date (Example: "2017-12-27 20:30:15" or 2017-12-27).
    --limit LIMIT         Number of Tweets to pull
    --near NEAR           Find tweets near a particular location
    --geocode GEOCODE     Search for geocoded Tweets.
    --year YEAR           Filter Tweets before specified year.
    --today               Filter Tweets from today
    --verified            Display Tweets only from verified users (Use with -s).
    --links LINK          Exclude tweets containing one or more links.
    --videos              Display only Tweets with videos.
    --images              Display only Tweets with images.
    --media               Display Tweets with only images or videos.
    --minLikes MINLIKES   Minimun likes for the tweet
    --minRetweets MINRETWEETS
                          Minimun retweets for the tweet
    --minReplies MINREPLIES
                          Minimun replies for the tweet
    --json JSON           File to write the JSON output to.
    --csv CSV             To store the output in CSV
    --silent              Dont print the tweets(Only works while taking an output!)

achyuthjoism-MBP ~ %

๐Ÿ“™ Example

CLI Example

Some simple examples to help you understand the basics:

  • tweeds -u username - Scrape all the Tweets of a user (doesn't include retweets but includes replies).
  • tweeds -u username -s pineapple - Scrape all Tweets from the user's timeline containing pineapple.
  • tweeds -s pineapple - Collect every Tweet containing pineapple from everyone's Tweets.
  • tweeds -u username --since "2015-12-20 20:30:15" - Collect Tweets that were tweeted since 2015-12-20 20:30:15.
  • tweeds -u username --since 2015-12-20 - Collect Tweets that were tweeted since 2015-12-20 00:00:00.
  • tweeds -s "Rocking Star Yash" --verified - Display Tweets by verified users that Tweeted about Rocking Star Yash.
  • tweeds --geocode "48.880048,2.385939,1km" --csv paris.csv --limit 10 - Scrape Tweets from a radius of 1km around a place in Paris and export them to a csv file.
  • tweeds -u username --images - Scrape Tweets from a user containing only images.
  • tweeds -u username --videos - Scrape Tweets from a user containing only videos.
  • tweeds -u username --media - Scarape Tweets from a user containing both images and videos.
  • tweeds -u username --links - Scrape Tweets from a user which excludes links.
  • tweeds -u username --json file.json - Scrape Tweets and save as a json file.
  • tweeds -u username --csv file.csv - Scrape Tweets and save as a csv file.

Module Example

Now Tweed can be used as a module and allows custom formatting. Example:

import tweeds

query = tweeds.Query()

query.search = "Yash Boss"
query.limit = 10
query.verified = True

tweeds.search(query)

Have fun ๐Ÿฅฐ๐Ÿ’ž

๐Ÿ“ฎ Details

Obvious disclaimer

This tool is for educational purposes only, I am not responsible for its use.

Less obvious disclaimer

This project is under MIT Licence, and you have to respect it.
Use it only in personal, criminal investigations, pentesting, or open-source projects.

tweeds's People

Contributors

achyuthjoism avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

tweeds's Issues

Order not found

โ”Œโ”€โ”€(kaliใ‰ฟkali)-[~]
โ””โ”€$ tweeds --help
tweeds: Order not found

Don't run on GitPod

I'm testing the code on Gitpod and it throws me several errors:

`Error retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=%28from%3AFuturoVegetal%29+&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe: blocked (403)

4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=%28from%3AFuturoVegetal%29+&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe failed, giving up.
Errors: blocked (403), blocked (403), blocked (403), blocked (403)

Traceback (most recent call last):
File "/workspace/.pyenv_mirror/user/3.11.1/bin/tweeds", line 8, in
sys.exit(main())
^^^^^^
File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/tweeds/main.py", line 117, in main
process_args(args)
File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/tweeds/main.py", line 51, in process_args
search(q)
File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/tweeds/tweet_scrape.py", line 76, in search
for i, tweet in enumerate(api.TwitterSearchScraper(query).get_items()):
File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/snscrape/modules/twitter.py", line 1546, in get_items
for obj in self._iter_api_data('https://api.twitter.com/2/search/adaptive.json', _TwitterAPIType.V2, params, paginationParams, cursor = self._cursor):
File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/snscrape/modules/twitter.py", line 759, in _iter_api_data
obj = self._get_api_data(endpoint, apiType, reqParams)

File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/snscrape/modules/twitter.py", line 725, in _get_api_data
r = self._get(endpoint, params = params, headers = self._apiHeaders, responseOkCallback = self._check_api_response)

File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/snscrape/base.py", line 225, in _get
return self._request('GET', *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/current/lib/python3.11/site-packages/snscrape/base.py", line 221, in _request
raise ScraperException(msg)

snscrape.base.ScraperException: 4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=%28from%3AFuturoVegetal%29+&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe failed, giving up.`

Cant you help me or explain me? I have reinstalled using 'pip install tweeds' and still get the same errors

blocked scraping

Hi, can you check the code? It seems the scraping is being blocked

Issue with since when attempting to filter with hours (--since)

When you type tweeds -s "node" --since "2022-12-30" --json file.json it works perfectly.

But adding the time to the --since parameter like this : tweeds -s "node" --since "2022-12-30 12:00:00" --json file.json Nothing is given(0 Results).

I believe there is an issue with the hours condition.

scraper not working

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.