kigamittsu / scrape_twitter

scrape_twitter

This API can search the tweet information from twitter by hashtag or username.

No need authentication

Prerequisites

If you want to use this API in local, please install the packages using pip or other package manager.

this application depends on the following libraries.

Library	version
flask	1.0.2
flask_restful	0.3.7
requests	2.21.0
selenium	3.141.0
chromedriver-binary	73.0.3683.68.0
gunicorn	19.9.0

Note: You have to download the chromedriver-binary which has same major version of your chrome browser. If you want to check the chrome browser version, you can check it in your browser

The following examples is using pip to install the dependencies.

1. Install the Flask.

pip install flask

2. Install the flask_restful

pip install flask_restful

3. Install the requests

pip install requests

4. Install the selenium

pip install selenium

5. Install the chromedriver-binary

pip install chromedriver-binary=="{Your Chrome version}"

6. Install the gunicorn

pip install gunicorn

Usage for API

1. Create the clone of project from github.

git clone https://github.com/kigamittsu/scrape_twitter.git

2. Run the server

cd ~/scrape_twitter
gunicorn server:app

3. Now you can call this API.

Testing API (Local)

1. Run the server

gunicorn server:app

2. Run the test

python -m unittest tests.test_api

API Design

/hashtags/{hashtagname}?{limit}

Get the tweets information which has target hashtag name

Resource

Resource	Description
hashtagname	Target hashtag name

Request parameter

Request Parameter	Type	Description
limit (Optional)	Integer	Specifies the number of tweets to retrieve, the default is 30

/users/{user}?{limit}

Get the tweets information which has target user name

Resource

Resource	Description
username	Target user name

Request parameter

Request Parameter	Type	Description
limit (Optional)	Integer	Specifies the number of tweets to retrieve, the default is 30

Sample Requests

If you want to test this API, we highly recommend to use the Postman or Insomnia

Sample requests of /hashtags/{hashtagname}

http://127.0.0.1:8000/hashtags/twitter?limit=20

Sample requests of /users/{username}

http://127.0.0.1:8000/users/twitter?limit=20

Sample Response

[
  {
    "account": {
      "fullname": "foo bar",
      "href": "https://twitter.com/foobar",
      "user_id": "342384"
    },
    "date": "16:23 - 2019年5月15日",
    "hashtags": [
      "#Twitter"
    ],
    "text": "Sample text",
    "likes": 10,
    "replies": 4,
    "retweets": 5
  },
  ...
]

Feature Plan

1. Improve response speed (more quick)
2. Add other endpoints
3. Write more test

License

This project is licensed under the MIT License