Twitter Celebrity Matcher

The objective of this project is to match celebrity users with their respective tweets by making use of Semantic Textual Similarity.

This app finds similar Twitter users based on their tweets. It works in two ways -

Get a list of the most similar celebrity Twitter accounts based on a predefined Twitter celebrity list (915
Twitter celebrity accounts).
Find similarities between two Twitter users based on their tweets.

View this project on kaggle - kaggle.com/code/ahmedshahriarsakib/twitter-celebrity-matcher-sbert-tweepy

💻 Streamlit App	☑️ How it works?	🛠️ Project Structure	💾 Dataset	📷 Screenshots	🔎 Findings	⬇️ Built With

Setup 🔧

Add the following to your .env file to work with tweepy:

ACCESS_KEY=YOUR_ACCESS_KEY
ACCESS_SECRET=YOUR_ACCESS_SECRET
CONSUMER_KEY=YOUR_CONSUMER_KEY
CONSUMER_SECRET=YOUR_CONSUMER_SECRET

Virtual Environment Setup

Set up a virtual environment (preferably python 3.8+, features such as walrus operator was used) and run:

$ pip install -r requirements.txt

Check main.py file to run the streamlit app or API.

Docker Setup

Build -

docker build -t twitter-celebrity-matcher .

Run -

docker run -p 8501:8501 twitter-celebrity-matcher

If you have CUDA enabled GPU, you can set up pytorch (for SBERT) with pip -

$ pip install --no-cache-dir --force-reinstall torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio===0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

Static type testing -

$ mypy main.py --ignore-missing-imports

Streamlit App💻

The app file is located at app/app.py.

To run the app, run the following command after installing necessary dependencies from requirements.txt -

$ streamlit run main.py

will start the app at default port 8501.

App Walkthrough

↥ Back To Top

Project Structure

TwitterCelebrityMatcher
├── api
│   ├── __init__.py
│   ├── api.py
├── app
│   ├── __init__.py
│   ├── app.py
|   ├── appdata.py
│   ├── apphome.py
│   ├── appinit.py
│   ├── utils.py
├── celebrity-listing
│   ├── Top-1000-Celebrity-Twitter-Accounts.csv
├── core
│   ├── __init__.py
│   ├── dataprep.py
│   ├── matcher.py
│   ├── scraper.py
│   ├── utils.py
├── twitter-celebrity-embed-data
│   ├── twitter-celebrity-embed-data.csv
└── .env
    |
    config.py
    |
    main.py

api - fastApi script
app - streamlit app scripts
- app.py - main app script
- appdata.py - dataclass contains configuration variables
- apphome.py - core functionalities
- appinit.py - helper script to initialize the core objects (scraper and matcher)
- utils.py - utility functions (e.g. social sharing html script)
core - contains 3 helper scripts
- dataprep.py - scripts for all preprocessing tasks ( tweet cleaning, embedding generation etc.)
- matcher.py - core functionalities to match twitter users
- matcher.py - twitter scraper helper script
- utils.py - utility functions
config.py - all configuration variables
main.py - run 3 apps - console scripts, streamlit app or API

utitlities directory contains a standalone tweepy scraper scripts - tweepy_scraper.py and emoticon proprocessing scripts (original from ekphrasis module)

How it works?

The celebrity tweets were collected using tweepy. After preprocessing the tweets, the tweets were embedded using sentence-transformers pretrained multilingual model - paraphrase-multilingual-MiniLM-L12-v2. The similarity score between tweets embeddings is calculated using cosine similarity.

This app -

Takes a Twitter username
Scrapes the tweets if it's unavailable
Generate embeddings of the given Twitter account's tweets
Calculate the mean embedding of the tweets (limitation ~3200 tweets)
Finds the cosine similarity between
- Given user's tweets embeddings and celebrity tweets embeddings
- Given two user's tweets embeddings
Returns the most similar celebrity Twitter accounts based on similarity score or just score between two users

Dataset

List of Celebrity Twitter Accounts

Open Source GitHub Gist - Top-1000-Celebrity-Twitter-Accounts.csv

NB:

There are some duplicates in the dataset. (986 after filtering)
There are some unofficial Celebrity accounts (ex - twitter.com/sonunigam) with a very small amount of tweets. We can filter those users based on their tweet count. Here is a good research paper on this topic - 25 Tweets to Know You: A New Model to Predict Personality with Social Media

I uploaded the relevant datasets on kaggle. Feel free to use the datasets -

Generated Embeddings of Celebrity Tweets

The final embeddings of all celebrity tweets -

Twitter Celebrity Embed Data

Data Preprocessing

The tweets were collected and saved as bytestring to preserve the text encoding (some tweets may contain complicated symbol/emojis etc.). So first the tweets are decoded as string and then some preprocessing techniques are applied.

Current pipeline :

Removed hashtags, urls, and mentions
Replaced emoticons with their textual representation (ekphrasis)
Replaced emoji with their textual representation (demoji package)

↥ Back To Top

Model

sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

Citation

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "http://arxiv.org/abs/1908.10084",
}

Word embedding dimension : 384

Download SBERT pretrained models directly from here

Screenshots

Match Top Celebrities

Match 1v1 Celebrities

User Interface

Result Section

Findings!🔎

Most of the celebrities are from music, film or sports industry. The similarity results in these categories are very impressive.

Music

Click to view similar celebrities from music industry - individual (query - Taylor Swift)

Click to view similar celebrities from music industry - band (query - Coldplay)

Sports

Click to view similar celebrities from music industry - footballer (query - Cristiano Ronaldo)

Regional Film Industry

The multilingual model does a great job finding region/culture specific attributes. Below is an example of bollywood celebrities

Click to view similar celebrities from film industry - actor (query - Shah Rukh Khan)

Author

The number of authors in this celebrity list is comparatively lower than other dominating professions (actors, singers, footballers etc.). So in this example the first few results (first two from the results -> both are above 0.94 score ) were good but then the score drops. The drop rate is higher that the results from the dominating celebrity professions where the top score is around 0.96+ and first 10 or 20 from the list have very close score. This will vary but with only 5 users the performance of the pretrained model is great in this case

Click to view similar celebrity authors - (query - John Green)

Multiple Roles - Business Magnet/ Philanthropist

From the screenshot below, it's very clear that there are few people on the list who are very active in multiple categories.
- The closest of them all (for Bill Gates) is Richard Branson although the score is comparatively lower (0.82) than what we see in other categories. Considering how close their roles are (both are billionaires, investor, tech) this is a very positive result.
- Leonardo DiCaprio is the second placeholder because he falls into Philanthropist category, same goes for Tony Robbins
- Other famous billionaires such as - anand mahindra and Ratan N. Tata makes it to the top 10
- Most of the top people are either founders of large organizations, activists, philanthropists or authors. (Bill Gates published few books, so this could be another factor)

Click to view similar celebrity billionaires/investors/businessmen/investor - (query - Bill Gates)

NB: Due to space limitation, the screenshots above displays only top 10 results, try the app to view more top results which are very similar to the query

↥ Back To Top

FYI

Test Workstation (RAM 16gb 2400Hz, AMD 2600x, 1050ti (768 CUDA cores))

With CUDA enabled GPU the app runs ~5x faster than CPU.
With the current scraping scripts, it takes ~14-16 seconds to download tweets
Takes ~2h 5m in total to preprocess and generate embedding for all users' (915 total) tweets

Built With

demoji==1.1.0
ekphrasis==0.5.1
jupyter==1.0.0
python-dotenv==0.19.2
schedule==1.1.0
sentence-transformers==2.2.0
torch==1.11.0+cu113
torchaudio==0.11.0+cu113
torchvision==0.12.0+cu113
streamlit==1.7.0
tweepy==4.7.0

ahmedshahriar / twittercelebritymatcher Goto Github PK