Coder Social home page Coder Social logo

dantegpt / twxiv Goto Github PK

View Code? Open in Web Editor NEW

This project forked from so-okada/twxiv

0.0 0.0 0.0 630 KB

This app twXiv gives arXiv daily new submissions by tweets, abstracts by replies, cross-lists by retweets, and replacements by quotes and retweets. We use python3 scripts. twXiv is not affiliated with arXiv.

Home Page: https://github.com/so-okada/twXiv

License: GNU Affero General Public License v3.0

Python 100.00%

twxiv's Introduction

Application Info

This application twXiv gives arXiv daily new submissions by tweets, abstracts by replies, cross-lists by retweets, and replacements by quotes and retweets. We use python3 scripts. twXiv is not affiliated with arXiv.

Setup

  • Install pandas, ratelimit, semanticscholar, tweepy, twitter-text-parser, nameparser, feedparser, and beautifulsoup4.

     % pip3 install pandas ratelimit semanticscholar tweepy twitter-text-parser nameparser feedparser beautifulsoup4

  • Let twXiv.py be executable.

     % chmod +x twXiv.py

  • Put the following python scripts in the same directory.

    • twXiv.py
    • twXiv_post.py
    • twXiv_format.py
    • twXiv_daily_feed.py
    • twXiv_semantic.py
    • arXiv_feed_parser.py
    • variables.py

  • Configure switches.json, logfiles.json, and aliases.json in the tests directory for your settings.

    • accesses.json specifies twitter access keys and whether to use new submissions/abstracts/cross-lists/replacements by twXiv.

    • logfiles.json indicates log file locations for tweet summaries, tweets, retweets, unretweets, and replies. You can check their formats by mathACb_tweet_summaries.csv, mathACb_tweets.csv, mathACb_retweets.csv, mathACb_unretweets.csv, mathACb_replies.csv, and mathACb_quotes.csv in the tests/logfiles director. twXiv needs a tweet log file for cross-lists and replacements. Other log files are useful to avoid duplication errors of tweets and retweets.

    • aliases.json tells twXiv aliases of arXiv category names. For example, math.IT is an alias of cs.IT. Without this file, twXiv of rss feeds returns no new submissions, when you take the category name math.IT. If provided, twXiv replaces category names by their aliases for new submissions, cross-lists, and replacements.

  • Configure variables.py for your settings.

    • variables.py assigns format parameters for twXiv tweets and access frequencies for arXiv and twitter.

Notes

  • arXiv_feed_parser.py is a simple arXiv feed parser for twXiv. We use this via twXiv_daily_feed.py to regularly obtain data.

  • Outputs of twXiv can differ from arXiv new submission web pages. First, this can be due to bugs in my scripts or connection errors. Second, items of arXiv rss feeds are not necessarily the same as those of arXiv new submission web pages (see https://mastoxiv.page/@vela/109829294232368163). Third, arXiv_feed_parser for an arXiv category C gives new submissions whose primary subjects are the category C. Then, twXiv for the category C counts and tweets a new paper whose principal subject matches the category C.

    • For example, let us assume that there is no new paper whose principal subject matches the category C, but there is a new paper P whose non-principal subject matches the category C. Then, the arXiv new submission web page of the category C lists the paper P as a new submission of the category C, not as a cross-list. However, twXiv keeps considering the paper P as a cross-list for the category C. Then, the output of twXiv for the category C differs from the arXiv new submission web page of the category C.

    • One reason for this is to clarify what twXiv tweets, retweets, and quotes across categories. Another reason is that twitter does not like duplicate or substantially similar tweets from bots. So, twXiv maintains a single tweet for the title, authors, and abs/pdf identifiers of a new paper across bots in access keys. Furthermore, if configured, twXiv tries to retweet and quote for cross-lists and replacements from bots in access keys.

  • On the use of metadata of arXiv articles, there is the web page of Terms of Use for arXiv APIs. As of the revision 0.8.0, this says that "You are free to use descriptive metadata about arXiv e-prints under the terms of the Creative Commons Universal (CC0 1.0) Public Domain Declaration." and "Descriptive metadata includes information for discovery and identification purposes, and includes fields such as title, abstract, authors, identifiers, and classification terms."

Usage

% ./twXiv.py -h
usage: twXiv.py [-h] --switches_keys SWITCHES_KEYS
                [--logfiles LOGFILES] [--aliases ALIASES]
                [--captions CAPTIONS] [--mode {0,1}]

arXiv daily new submissions by tweets, abstracts by
replies, cross-lists by retweets, and replacements by
quotes and retweets.

optional arguments:
  -h, --help            show this help message and exit
  --switches_keys SWITCHES_KEYS, -s SWITCHES_KEYS
                        output switches and api keys in
                        json
  --logfiles LOGFILES, -l LOGFILES
                        log file names in json
  --aliases ALIASES, -a ALIASES
                        aliases of arXiv categories in
                        json
  --captions CAPTIONS, -c CAPTIONS
                        captions of arXiv categories in
                        json
  --mode {0,1}, -m {0,1}
                        1 for twitter and 0 for stdout
                        only

Sample stdouts

  • New submissions for math.AC and math.AG with no log files:

     % ./twXiv.py -s tests/switches.json -m 1
 **process started at xxxx-xx-xx xx:xx:xx (UTC)
 starting a thread of retrieval/new submissions/abstracts for math.AC
 getting daily entries for math.AC
 waiting for a next thread of retrieval/new submissions/abstracts
 new submissions for math.AC
 no log files
 no log files

 utc: xxxx-xx-xx xx:xx:xx 
 thread arXiv category: math.AC
 arXiv id: 
 url: https://twitter.com/user/status/
 post method: tweet
 post mode: 1
 url: https://twitter.com/user/status/xxxxxxxxxxxxxxxxxxxxxxxx
 text: [xxxx-xx-xx Sun (UTC), 2 new articles found for mathAC]

 starting a thread of retrieval/new submissions/abstracts for math.AG
 getting daily entries for math.AG
 joining threads of retrieval/new submissions/abstracts

 utc: xxxx-xx-xx xx:xx:xx
 thread arXiv category: math.AC
 arXiv id: xxxx.xxxxx
 url: https://twitter.com/user/status/
 post method: tweet
 post mode: 1
 url: https://twitter.com/user/status/xxxxxxxxxxxxxxxxxxxxxxxx
 text: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

 ....

 **process ended at xxxx-xx-xx xx:xx:xx (UTC)
 **elapsed time from the start: xx:xx:xx

  • New submissions, abstracts, cross-lists and replacements (if any) for math.AC and math.AG:

     % ./twXiv.py -s tests/switches.json -l tests/logfiles.json -a tests/aliases.json -c tests/captions.json -m 1
 **process started at xxxx-xx-xx xx:xx:xx (UTC)
 starting a thread of retrieval/new submissions/abstracts for math.AC
 getting daily entries for math.AC
 waiting for a next thread of retrieval/new submissions/abstracts
 new submissions for math.AC

 utc: xxxx-xx-xx xx:xx:xx 
 thread arXiv category: math.AC
 arXiv id: 
 url: https://twitter.com/user/status/
 post method: tweet
 post mode: 1
 url: https://twitter.com/user/status/xxxxxxxxxxxxxxxxxxxxxxxxx
 text: [xxxx-xx-xx Sun (UTC), 2 new articles found for mathAC Commutative Algebra]

 starting a thread of retrieval/new submissions/abstracts for math.AG
 getting daily entries for math.AG
 joining threads of retrieval/new submissions/abstracts

 utc: xxxx-xx-xx xx:xx:xx
 thread arXiv category: math.AC
 arXiv id: 2007.xxxxxx
 url: https://twitter.com/user/status/
 post method: tweet
 post mode: 1
 url: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

 new submissions for math.AG

 utc: xxxx-xx-xx xx:xx:xx
 thread arXiv category: math.AG
 arXiv id: 
 url: https://twitter.com/user/status/
 post method: tweet
 post mode: 1
 url: https://twitter.com/user/status/xxxxxxxxxxxxxxxxxxxxxxxxxxxx
 text: [xxxx-xx-xx Sun (UTC), 4 new articles found for mathAG Algebraic Geometry]

 utc: xxxx-xx-xx xx:xx:xx
 thread arXiv category: math.AC
 arXiv id: 2007.xxxxxx
 url: https://twitter.com/user/status/xxxxxxxxxxxxxxxxxxxx
 post method: reply
 post mode: 1
 url: https://twitter.com/user/status/xxxxxxxxxxxxxxxxxxxx
 text:  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [1/4 of https://arxiv.org/abs/2007.xxxxxv1]

 .....
 
 **crosslist process started at xxxx-xx-xx xx:xx:xx (UTC) 
 **elapsed time from the start: xx:xx:xx
 start a crosslist thread of math.AC
 waiting for a next crosslist thread
 start a crosslist thread of math.AG
 joining crosslist threads

 **replacement process started at xxxx-xx-xx xx:xx:xx (UTC)
 **elapsed time from the start: xx:xx:xx
 **elapsed time from the cross-list start: xx:xx:xx
 quote-replacement starts
 start a quote-replacement thread of math.AC
 waiting for a next quote-replacement thread
 start a quote-replacement thread of math.AG
 
 .....
 
 utc: xxxx-xx-xx xx:xx:xx
 thread arXiv category: math.AG
 arXiv id: 2010.xxxx
 url: https://twitter.com/user/status/xxxxxxxxxxxxxxxxxxxx
 post method: quote
 post mode: 0
 url: https://twitter.com/user/status/
 text: This https://arxiv.org/abs/xxxx.xxxx has been replaced.....

 .....

 retweet-replacement starts
 start a retweet-replacement thread of math.AC
 waiting for a next quote-replacement thread
 start a retweet-replacement thread of math.AG
 
 utc: xxxx-xx-xx xx:xx:xx 
 thread arXiv category: math.AG
 arXiv id: 1900.xxxxx
 url: https://twitter.com/user/status/xxxxxxxxxxxxxxxxxxxx
 post method: unretweet
 post mode: 1
 url: https://twitter.com/user/status/xxxxxxxxxxxxxxxxxxxx
 text: 

 utc: xxxx-xx-xx xx:xx:xx 
 thread arXiv category: math.AG
 arXiv id: 1900.xxxxx
 url: https://twitter.com/user/status/xxxxxxxxxxxxxxxxxxxx
 post method: retweet
 post mode: 1
 url: https://twitter.com/user/status/xxxxxxxxxxxxxxxxxxxx
 text: 

 .....

 **process ended at xxxx-xx-xx xx:xx:xx (UTC)
 **elapsed time from the start: xx:xx:xx
 **elapsed time from the cross-list start: xx:xx:xx
 **elapsed time from the replacement start: xx:xx:xx

  • Without the option -c tests/captions.jsonabove, you get

     text: [xxxx-xx-xx Sun (UTC), 4 new articles found for mathAG]

    instead of

     text: [xxxx-xx-xx Sun (UTC), 4 new articles found for mathAG Algebraic Geometry]

Versions

  • 0.0.1

    • 2020-08-08, initial release.

  • 0.0.2

    • 2020-08-15, added the -c option.

  • 0.1.0

    • 2021-01-31, text counting by twitter-text-parser, replacements by quotes and retweets, and links to research tools.

  • 0.1.1

    • 2021-03-02, minor fixes.

  • 0.1.2

    • 2021-06-03, a fix for semantic scholar api update.

  • 0.1.3

    • 2021-10-26, a fix for tweepy v4.

  • 0.1.4

    • 2022-08-20, minor fixes.

  • 0.1.5

    • 2022-01-09, a fix for semanticscholar 0.3.2.

  • 0.2.0

    • 2023-06-25, a fix for twitter API V2 endpoints and a daily rate limit.

List of Bots

Author

So Okada, [email protected], https://so-okada.github.io/

Motivation

Since 2013-04, the author has been running the bots above for all arXiv math categories. Until 2020-08, the author used arxiv_speaker https://github.com/misho104-obsolete/arxiv_speaker. Sho Iwamoto wrote it by ruby for tweeting new submissions with titles, authors, and ids. However, it was discontinued. Therefore, the author has written twXiv by python3, adding functions such as abstracts by replies, cross-lists by retweets, and replacements by quotes and retweets. Since 2020-08, the bots above have used twXiv on gcp free tier.

For more background, see a paper on these bots, bots of Notionsandnotes, and the author's comments:

I am running these bots mainly just for convenience of sharing wisdom of arXiv papers for each arXiv category.

— So Okada (@vela) May 9, 2016

This would end my 32 math arXiv twitter bots such as https://t.co/w8cCIaUlBM, https://t.co/2j62N9Vk8a, ... They have shared wisdom of arXiv papers with comments, discussions, or jokes not just to academics but to the public. I have maintained them nearly 10 years from 2013-04. https://t.co/J0WUSSTlwZ

— So Okada (@vela) February 5, 2023

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

AGPLv3

twxiv's People

Contributors

so-okada avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.