42-ai / sentimentalbb Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
Trying to incorporate an interactive Weight and Biases graph inside a markdown to make github-pages more user friendly
Just an exploration so no commit
Temporary set the SBB README as the website index page
Website index page = SBB README
echo "source venv/bin/activate" > .envrc
to echo "source venv/bin/activate" >
PATH TO VENV/ .envrc
source venv/bin/activate
to activate venvCreate python code under src/data
which is called from python -m src data --download user
Download them under data/raw/user/
. We keep ALL the data received by twitter API.
When tweeets are dowloaded to data/raw/user
Add my github and 42 logins to README.md
Edit README.md
Objective: Build the database and having data on as much days as possible.
Update the script about dataset to request specific tweets from the API of twitter based on its date or ID.
The script MUST save ALL the tweets received into csv
files in the data/raw/twitter
directory, with the date and ID of first and last tweet specified (in the name ?). Possible format:
data/raw/twitter/[candidat_name]_[startdate]_[enddate].csv
data/raw/twitter/[candidat_name]_[first_id_tweet]_[last_id_tweet].csv
data/raw/twitter/candidat_name/[startdate]_[enddate]_[first_id_tweet]_[last_id_tweet].csv
data/raw/twitter/week_#x/[candidat_name]_[startdate]_[enddate]_[first_id_tweet]_[last_id_tweet].csv
A particular point must be considered: the script should collect small chunks of results in order to save all the results little by little to avoid issue related to cache memory, disk memory or whatever: Create a tmp
directory where the little portions are stored and after that the script
This script should be designed to be launched periodically, every week (or every day?) and collect specified amount of tweets about each candidate. These amounts of tweets per day and per candidate are yet to be determined.
tmp
directory where it saves small chunks of the total results.csv
file.This script should be FIRST and ONLY tested with small amounts of tweet requested to the API in order to save the amount of tweet we can request: for instance 1k tweets for 2 or 3 candidates. The person testing the script should be careful to check the above points.
This script will be used for larger amounts after the pull request is validated.
Add my name and 42 login to the README.md
Name and login added to README.md
This is a test
Fix the error that happens when executing the script for downloading tweets about macron:
cannot read file ~/.twitter_keys.yaml Error parsing YAML file; searching for valid environment variables
No error happens when executing python -m src data --download macron
Update the website template
Add my name and 42 login to the readme.md
Update the list of contributors
readme.md should have my name and login
Write a very small documentation of the function src/data/make_dataset/macron.py
Documentation is written.
Make a deeplearning baseline with Hugging Face and Pytorch.
Relevant tutorials to follow:
Relevant documentation:
poetry run python -n src models --model camembert --output-weights models/xxxxx --train-split xxx.csv
dvc add models/xxxxx
git add && git commit
dvc push -r s3-remote
src/tests/test_dataset.csv)
.github/workflows/cicd.yaml
and the .42AI/pre-commit.git
to pass the new unit testUpdate the Setup.md
file to remove the use of direnv and .envrc
file.
Also remove the error when executing .42AI/init.sh
after cloning the repo.
Setup.md
is updated and .42AI/init.sh
raises no error.
The describa encapsulate everything
A clean setup.md when viewed on github
Create a directory dedicated to GitHub Pages + index template
Available online index template page
While waiting for our collected twitter dataset to contain the datetime of each tweet we would like to plot our predicted sentiment for each candidate.
There are multiple otpions:
For each candidate, any figure from the top list saved under reports/figures/candidate/type_of_chart_and_name_question_answered.png
ie: if it's a Pie Chart for Macron reports/figures/Macron/pie_chart_sentiment_prediction.png
This issue must be completed only after issue #75 is completly done.
Run the feature built for issue #75 to download locally tweets from twitter API.
Make sure you have enough place available and your machine works correctly before doing so.
Then push these tweets with DVC to the AWS S3 storage service.
Determine the amount of tweet to request per day and per candidate after consultation with the others members of the project.
You should download tweets for one candidate with the start_date: 7 days ago and end_date: 6 days ago.
The .csv
file must have at least this start_date and the last tweet ID in its name, like advised in issue #75, in order to be able to complete the data for this day and candidate later if possible.
This dataset should be reproductible : you must save somewhere the query sent to the Twitter API, either in the .csv
file or in another file dedicated to it. The idea is to be able to publish this dataset on thr Huggingface Hub, and it requires reproductibility.
Tweets for ONE candidate and ONE day in the right amount regarding our request capacity are saved in the data/raw/twitter
directory (or data/raw/twitter/week_#x/
) and pushed to the DVC remote storage.
The script should suggest the user to perform the commands to add and push the data obtained to the dvc remote.
Every data obtained from the twitter API should be pushed to the dvc remote.
The query used to obtain the tweet must be saved.
When adding dvc parts: add the folders under data/raw
and data/processed
separately
Able to dvc push the latest version of data to the remote s3 AND being able to pull it
Able to use DVC data in github action:
ie:
dvc pull
to .42AI/int.sh
Add my github and 42 logins to the README.md
Edit the README.md
Readme should have a short description of the project
Readme should have a table of content see here
Readme should have the following parts
reports/figures/
)poetry install
) such as to predict on tweets (poetry run ...
)Setup.md
and CONTRIBUTING.md
)
All the listed element appears on the Readme.md when visiting https://github.com/42-AI/SentimentalBB
In Setup.md
add steps to connect to remote s3, and how to add
/push
/pull
data
In .42AI/init.sh
dvc pull
When steps in Setup.md
allow new collaborators to participate, and when .42AI/init.sh
gets the latest version of data available
Adding of my login github and 42 in the README.md
Edition of the README.md.
In .github/ISSUE_TEMPLATE
add in the top .yaml part project: SentimentalBB
status: Backlog
When a new issue is created, it does not have the status "No status" and it is linked by default to SentimentalBB
This issue is due to 42 workspace consideration. B y default poetry is install in ~/HOME/.poetry
.
Due to the limitation of the space, it could be interesting to be able to install Poetry in the sgoinfre
in our personal repository /sgoinfre/
. We could also create a repository 42AI_sbb
into the sgoinfre to share the setup of the project.
poetry run python -m src ...
withpoetry installled into the sgoinfresgoinfre
While waiting for the allocine training dataset from the Milestone 3 let's train a model with our temporary dataset.
The temporary dataset is the aclImdb
Standford datasaet.
sh scripts/download_dataset.sh
python -m src data --download aclImdb
The 1st model:
When you have your predicted results saved under data/processed/aclImdb/results/classifier_name.csv
Your results should at least contain: y_pred
and y_true
(as we are using the training dataset, we have access to the correct label associated with each X)
There should should be a unit test which launches the train and predict pipeline, and then calculate the accuracy from this saved file.
For each tweet saved, add the datetime information
When the dataset created under data/raw/twitter/macron/twitter_Macron_None_None.csv
will contain datetime information
Add a feature that requests the number of tweets published on twitter for a given candidate and a given day.
You may also request in addition the number of tweets for each hour of that day.
You should maybe use the granularity
parameter of the function that makes querys in searchtweet-v2.
These data will help us decide how much tweet per candidate we should request, and have an overview of the political activity on twitter that we can display on a graph.
This data should be saved the somewhere data
folder, maybe add a new data/metrics
directory. Then push it to the dvc remote.
In ordre for this data to be reproductible, you should also save the request you made to obtain these answers from twitter API. You can either write the query in the same file or in another file.
Add a feature to apply a model defined in src/models
to a .csv
file with raw data stored in the data/raw/
directory.
Save the output in a .csv
file in data/processed/
, from the command line.
The new file should contain the same name as the raw data file.
Feature is done.
Then apply this feature to the data already collected.
I want to predict results with a given model on a given csv
saved results under path/results.csv
Train Model Baseline with HuggingFace allocine dataset https://huggingface.co/datasets/allocine
Able to pass naive-bayes unit test
reports/figures/
and a template to describe the result of the experiments.reports/figures/
reports/
Setup.md
such as there is the correct steps to setup the environmentpyhton -m src data --download twitter --mention Macron
Create a virtualenv
python3.8 -m venv venv
The virtual environment was not created successfully because ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
python3.8 -m virtualenv venv
Having a github pages which diplay one image from the repository (for example: .42AI/assets/Step_1.png
)
Able to send an url to another person which shows a webpage with an image in it
Contributing.md
or similar where one can find how to contribute.Contributing.md
or similar where one can find how to contribute.Update the dev issue template to add the project SentimentalBB
by default, into the BackLog
status by default also.
When creating a issue with the dev template, the project and status are set by default.
Create a specification document detailing everything that the program contained in src is supposed to do.
Create python code under src/data
which is called from python -m src data --download --user username --date-from 01/01/2020 --date-to 01/02/2020
Download them under data/raw/user/
. We keep ALL the data received by twitter API.
When tweeets are dowloaded to data/raw/user
Register a file in src/data, that will download a tweet which mention Emmanuel macron and saves it in data/raw/twitter/user/
When a tweet will be registered in above folder by running a script in src/data
Please answer the following questions for yourself before submitting an issue. YOU MAY DELETE THE PREREQUISITES SECTION.
Please describe the behavior you are expecting
What is the current behavior?
Please help provide information about the failure if this is a bug.
If it is not a bug, please remove the rest of this template.
Please provide detailed steps for reproducing the issue.
Please provide any relevant information about your setup.
This is important in case the issue is not reproducible except for under certain conditions.
Please include any relevant log snippets or files here.
install poetry
worflow completes on master
Add model twitter-xlm-roberta-base-sentiment from carfifnlp shared on Huggingface.
.github/workflows/cicd.yaml
end the .42AI/precommit.git
to pass the new unit test.A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.