fanfictionbot / reddit-ffn-bot

A reddit auto reply bot.

FanFiction Bot for Reddit

This bot was made to allow reddit users to automatically get their responses, requests and promotions linked with the title, author and metadata of the story.

Features

• Direct Parsing of Links
• Modification of bot behaviour for the users.
• Support for fanfiction.net, fictionpress.com, hpfanficarchive.com, archiveofourown.org, siye.co.uk and adultfanfiction.com

Deployment and Installation

If you want to run your own bot or contribute to this project, make sure you have the following dependencies installed on your machine:

• Python 3.4 or newer

Install bot as follows: Adapt the commands for your linux distribution We assume Ubuntu 14.04 and you are inside your desired installation directory.

    $ sudo apt-get install git python3.4 lxml libxml2-dev python3.4-dev
    $ sudo apt-get install python-pip
    $ git clone https://github.com/tusing/reddit-ffn-bot .
    $ pip install -r requirements.txt

Cloudflare Bypass needed for Fanfiction.net

You must install and run the FlareSolverr container:

docker run -d \
  --name=flaresolverr \
  -p 8191:8191 \
  -e LOG_LEVEL=info \
  --restart unless-stopped \
  ghcr.io/flaresolverr/flaresolverr:latest

Configuring the bot

On your Reddit account, create an app (Preferences > Apps > Create Another App). Enter https://github.com/tusing/reddit-ffn-bot/wiki/Usage as the about url and http://127.0.0.1:65010/authorize_callback as the redirect uri. Hit create.

The value under personal use script is your client ID; the value next to secret is your client secret. Open up config.ini and replace the [REDACTED]s with the appropriate values (including your own username and password).

Replace the subreddits value with a comma-separated list of the subreddits you'd like to run the bot on.

Starting the bot

Use the following command to start the bot

    $ python . -d

Advanced usage:

usage: . [-h] [-s SUBREDDITS] [-d] [-l] [-v VERBOSITY] [-c CONFIG]

optional arguments:
  -h, --help            show this help message and exit
  -s SUBREDDITS, --subreddits SUBREDDITS
                        define target subreddits; separate with commas
  -d, --default         add config file subreddits, can be in addition to -s
  -l, --dry             do not send comments.
  -v VERBOSITY, --verbosity VERBOSITY
                        The default log level. Using python level states.
  -c CONFIG, --config CONFIG
                        The location of your config.ini.

Contributing

We happily accept contributions. Please note, that we only accept pull requests into the bleeding-branch. You can find our issue page here: Issues You can find our pull request page here: Pull Requests

reddit-ffn-bot's Issues

Minor Enhancement: Add .mobi download support; add FictionPress support

A minor change just involving changes to p0ody's download string provided to us at the end of #81.

Google backend will not work forever

When the bot receives a request, it searches for that request in Google (adding on, for example, "site:fanfiction.net" to the query if the request was for fanfiction.net).

The script we use to search Google will not work forever. Google doesn't really have an API and it's cracking down on script-based access. To ensure the stability of the bot, we'll want to use a sanctioned API-based method of crawling Google.

An option might be Google's Custom Search APIs, where we define Custom Search over the two or three websites we support. We would then replace our call to the Google script with a call to Custom Search.

Updated! notice is misleading

When I saw "Updated 7/19/15!" I thought it was referring to the story being updated. Perhaps you could clarify the text to indicate that it's the linkbot that updated rather than the story?

I think we should refactor the codebase of reddit_bot.py

I think, the codebase of reddit_bot.py grows a little bit large.

So I think, we should be splitting up the file into three parts:

• The part that sets up the reddit bot.
• The reddit-bot itself
• And the part that is actually forming the replies.

This makes it easier to add new things into the bot like

• Use threading to query and send reddit messages
• Handle multiple reddit messages simultaneously

If fanfiction has no image, no data is displayed

For fanfictions with no cover image, no data is displayed.

For example, this fanfiction doesn't have an image, so things like the word count, etc don't display in the final comment.

This is because the XPaths for the description change depending on the presence of an image.

AO3 Series Linking

It shouldn't be too hard to implement: take a look at the Island of Fire series on AO3, at the XPath //*[@id="main"]/ul:

<ul class="series work index group">
    <li class="work blurb group" id="work_3236603" role="article">
    <li class="work blurb group" id="work_3269186" role="article">
    <li class="work blurb group" id="work_3278756" role="article">

The story ID's all there for a series - just work_StoryID. We can go through these and we should have what we need.

Fanfiction bot won't show more than 1 post of fics (6 fics)

Lately, I have several cases where the bot shows only of the 6 fics requested in the same post, even though more are requested. It only posts one reply, containing 6 of the fics, and the rest are be forgotten.

For example, here (https://www.reddit.com/r/NarutoFanfiction/comments/829u5d/weekly_fic_request_thread_post_fic_requests_here/dvi15ld/) I requested 47 fics: one post with 25, and one post with 22 requests. Each post was answered with only a single post from the bot, containing 6 of the fics requested. I had to split the requests further into 6-request posts to get the bot to link them all.

Another similar case I have seen here: https://www.reddit.com/r/HPfanfiction/comments/7ymqb6/my_hermione_timetravel_compendium_of_the_last/. Out of all the fics the OP has requested, only 6 are actually linked.

[Enhancement] Update the docs?

ffnbot!autolink tag

Basically #6, but requires the tag ffnbot!autolink to activate.

The idea behind this is that people can make nicer-looking comments.

So instead of typing

blah blah blah
linkffn(ficname)

people can make their comment look nicer AND get a bot reply by doing this:

blah blah blah
a bunch of hyperlinks to fanfiction.net
^^^^^^^ffnbot!autolink

the benefits of which are

• Allows for nicer-looking comments
• Allows for users to create comments without visible references to the bot

Hyperlinks on mobile

The hyperlinks will not redirect on mobile platforms (or at least the official reddit app).

[Bug] Story instance is left in an invalid state when it fails to retrieve page.

Possible fix:

• Remove the try-except in Story.__init__

Sanity and Edit Checks

I think we should add two things:

• Sanity Checks will make sure that the bot has replied to every FanfictionBot call. This will require changing from using checked_comments.txt to checking the replies to comments calling the bot. If replies to a comment contain FanfictionBot as an author, the bot will not respond. Otherwise, the bot will create a reply.
• Edit Checks will update FanfictionBot replies if the parent comment has updated their submission.

A fundamental change in this is that we won't be using checked_comments.txt to make sure we've replied to a comment - instead, we'll check the replies to every comment that calls the bot, instead.

Incorrect links generated by linksub() function

When using the linksub function, it looks like it is using the old download links for the fics instead of generating a current version. This poses a problem because of #92. The links being output are no longer valid, pointing to the expired www.p0ody-files.com domain.

[Improvement] Add __pycache__ and CHECKED_COMMENTS.txt to .gitignore

Well acutally, add the python gitignore to your gitignore
See https://github.com/github/gitignore/blob/master/Python.gitignore

pause() not skippable

pause() in reddit_bot.py is not skippable. Looking for a pythonic way to implement a skippable pause function.

Different converter - P0ody Files

Have you ever used http://p0ody-files.com/ff_to_ebook/ as a converter? IMO it works better than FicSave, and takes no time if it has already been converted because it saves the fics it converts. And its formatting has the fix information as well.

Has been implemented but I will try and get in touch with P0ody to see if he can rig it up for FictionPress as well.

/u/StuxCrystal said it didn't work for him, but it works fine for me, /u/Psantium, and /u/tusing.

https://www.reddit.com/r/HPfanfiction/comments/3e57bb/fanfictionbot_usage_tips_updated_check_em_out_here/ctcv546

[Request Thread] Automate ffnbot!refresh for sufficiently new comments and submissions

Mentioned Here:
https://www.reddit.com/r/HPfanfiction/comments/3kmijc/ffnbot_request_a_feature/cuynx1g

Description:
Check for edits on sufficiently new comments. (MD5-Hashing comments)?

Adding another subreddit

Hi. I'd like to run FFNBot on the r/MiraculousFanfiction subreddit (see https://www.reddit.com/r/MiraculousFanfiction/comments/doykeh/should_this_sub_include_ffnbot/). Would you be open to adding that to the default list, or should I run a separate bot?

[Improvement] persistent_main is recursive

Persistent Main is not really persistent. Given enough exceptions a stack overflow will occur, causing the application to crash.

A loop may be more appropriate, so you can just print the exception before you restart the main function without having to call persistent_main recursively.

Parse story-metadata depending on site.

I think, it would be best to parse the metadata depending on the site returning a (eventually ordered) dictionary on a Story.get_stats call.

Reply from Author's Name

Another suggestion:

What about adding a way to take an author's name and have the bot reply with their profile/other names?

I don't know how hard this will be to implement, but I was thinking about this:

---

    authorffn(Kwan Li)   

Kwan Li has written 6 stories for Harry Potter:

• Hogwarts Battle School
• The Other Boy Who Lived
• The Dark of Night
• The Boy Who Killed
• [Wedding Dress](https://www.fanfictio. net/s/6326753/1/Wedding-Dress)
• My Name is Harry Potter

It's linked here: https://www.reddit.com/r/HPfanfiction/comments/3e57bb/fanfictionbot_usage_tips_updated_check_em_out_here/ctemom2

Some very important todos

Some milestones I really think we should be trying to achieve.

IO-Stuff (Very complex stuff in the current state of the bot)

• Switch to a threading model to continuously query reddit
• Switch from threading to asyncio (Has something to do with python interna See here for long explanation, what happens when threading is done wrong)

Other Stuff (Relatively easy to do)

• Use SQL-Database to efficiently store and query checked comments.

Bot messes up with parantheses or semicolons in fic title

Example:
linkffn(Applied Cultural Anthropology; Demonologist; Rise of the Dark Lady Hermione (adopted); Lord Hermione?; Divided and Entwined; Adorable Violence)
The bot only links the first 3 fics. What's probably happening is that the parsing stops after the first closing paranthesis. FFN doesn't allow semicolons in titles, but AO3 does, which causes further trouble. Fuzzy search allows the bot to still find the correct fic most of the time though.

Adding another subreddit

Hi. I'd like to run FFNBot on the r/HarryandGinny subreddit (see https://www.reddit.com/r/HarryandGinny). I have been running my own instance for this subreddit for a month or so. Would you be open to adding that to the default list?

Using as a standalone (non-posting) filter?

Would it be possible to add some switch to the script, so it would work as a standalone filter, which would process for example this input in reStructuredText:

perhaps the closest i got to it was linkffn(6432055), but even that was
more story of a pureblood trying to find his way in the strange world of
muggles then true redemption story of dealing with guilt. Nevertheless,
it is still a very good story.

into this output

perhaps the closest i got to it was `Exile by bennybear`_, but even that was
more story of a pureblood trying to find his way in the strange world of
muggles then true redemption story of dealing with guilt. Nevertheless,
it is still a very good story.

.. _`Exile by bennybear`:
   https://www.fanfiction.net/s/6432055   

(or an equivalent in Markdown, I don’t care that much)

Serverless Search

This would help to avoid any bans and keep the costs low too, probably. For GCE, this will mean Cloud Functions.

[Feature Request] Add semicolon for listing several stories inside single linkffn(...)

So, for instance, instead of

linkffn(ID01) linkffn(ID02) linkffn(ID03)

you’ll only need to use

linkffn(ID01; ID02; ID03)

Bot multi-posting copies of the same post several times

As seen here: https://www.reddit.com/r/NarutoFanfiction/comments/8b2h72/weekly_fic_request_thread_post_fic_requests_here/dx47pe0/

[Feature Request]: Asking the bot to summarise and recommend *threads*.

Many story requests repeat themselves once in a while, usually because OPs don’t search through the subreddit first or because a vague title/description combo made these threads harder to find.

I suggest adding an additional option for the bot to link and summarize subreddit threads as well.

---

I also had two other long-term feature reqs,^1 but I don’t know whether I should write about them here or open a new thread for each one.

^1:
b) asking the bot via PMs to reply to not litter the thread comments with twice as much comments,
c) archiving each story that gets downloaded (p0ody-files just converts them for downloading and discards them, AFAIK).

HPFFA metadata doesn't superscript properly for WIP fics

Issue is due to the fact that parentheses in strings grabbed by metadata end the superscript.

Ways to fix this could be:

• Escape all parentheses in metadata
• Filter out all parentheses when grabbing metadata
• Manually replace WIP (Work in Progress) with WIP

Issue can be seen in action here.

I can not do this at the moment as I am not near my computer.

Preserve fic linking order

The next major update should make it so order for fics is preserved - making linking a series of fics more useful. This should help if we ever decide to implement AO3 series.

Move to SQL

I have multiple things in mind that require SQL in some form or another, I suggest trying SQLAlchemy so we can get more than one choice of database systems, for production or development.

Ideas:

• Add code tp track feature usage
• Add code to allow users to permanently opt out or in
• Easier and more efficient data recall like which comments have already been seen.

More sites

@stuxcrystal @StuxSoftware I will add HPFanficArchive but I would rather do this myself, as a learning experience, so I probably won't look at any pull requests adding this site unless I ask for help on it. After I'm done, it's open season for pull requests regarding the addition for this site. Just in case you were also planning to do it, since you work way faster than I do 😄

Move string comment generators to __str__() functions

Title says it all. Should be cleaner and more standardized.

Edit: Just notice this is already done for ao3

linksiye doesn't seem to work

https://github.com/tusing/reddit-ffn-bot/wiki/Usage claims that linksiye() should work in the same manner as linkffn() and linkao3(), but it doesn’t. Either fix the bot, or fix the documentation.

Add capability to group stories

Add a group class whose feature is to post as many stories as possible under one post.
Each group will have a header that describes the group.

Requirement for #80 and #78

Private Message Responding

Many reddit bots also allow users to private message the bot to get response via private message.

Ao3 Author's Name Link

This appears to only effect Ao3 links.

It's missing the domain. It's linking only to '/users/rubypop/pseuds/rubypop' (example). So you end up at a 404.

See: https://www.reddit.com/r/FanFiction/comments/4y9f5f/the_best_fanfiction/d6pg9yu

[Feature Request] Automatically assume linkffn when a link is posted.

I don't think you requre linkffn when you post a link to a fanfiction.

PRAW 3 Support

Currently, the bot uses PRAW 2, which is deprecated and will stop working in just under a month (8/3/15). We need to switch to PRAW 3. which doesn't see, like an easy task at first glance.

PRAW 3 Changelog

A relevant requirement is:

• All requests should now be through HTTPS.

PRAW 3 allows for 2x faster requests, essentially doubling the speed of subreddit parsing.

No setup.py

[Feature Request] Include Fandoms in fic-data.

So you know if this is a Dagan Ropa fic or a HP fic. Or a crossover.

[Improvement] Use str.ljust for string formatting

See documentation here: https://docs.python.org/2/library/stdtypes.html#str.ljust

the bot should be case insensitive

See this thread: Ffnbot!parent didn't work, ffnbot!parent did. It so silly mistake to make, especially on a smartphone which have a tendency to automatically capitalize the first letter, that it should be best to follow the Postel’s rule and accept ffnbot commands withou caring about the case of the command.

ffnbot!directlinks doesn't always recognize multiple links?

As an example, I posted this comment

I'm only linking [*Hatal Fart Attack*](https://www.fanfiction.net/s/2814791/1/Hatal-Fart-Attack) because it operates as a prequel to the other fanfic I'm recommending, [*Royal Ward*](https://www.fanfiction.net/s/10556030/1/Royal-Ward). Mind you, unlike the other one I suggested, Sirius isn't the main character, but he does play a major role in the story later on. He does also date someone(I'm not quite certain they count as an OC?)

And the FFNBOT linked the first one, Hatal Fart Attack, but failed to link the second, Royal Ward

[Suggestion] Add a develop branch

The idea behind it is that we can distinguish which version is currently used by the actual bot and which one is the current development snapshot.

It works like this:
Whenever you update the VPS-Version you first merge master with develop and then pull from master.
With this you have a good backup of the currently used version

Directly mark for later/follow link?

Would it be possible to add links to the bot that automatically put a story on the "mark for later" list (ao3) or bookmark list (ffn)?
You can add /mark_for_later to an ao3 link to automatically add a fic to the list.
Adding a bookmark on ffn doesn't have an easy link system but the following code in a ffn page seems to do the trick: $.post('https://www.fanfiction.net/api/ajax_subs.php',{storyid: storyid,storyalert: 1 },'json'); where storyid is the id of the story.

[Converter] New domain

Hey guys, not sure where to put this so you see it ASAP but, i just realized that my domain & host (http://www.p0ody-files.com/ff_to_ebook/) expire on June 8th (I actually thought it was 6th August :P)

Anyway, im still working on my new version of my site, so i moved the old site to my new domain so you can still use it to (http://ff2ebook.com/old/).

You will just need to change you url in your bot.

Something like http://www.ff2ebook.com/old/ffn-bot/index.php?id=$ficID&source=$site&filetype=$epubORmobi

Cya

Related Works on AO3 make title search unusable

The bot gets confused by related works on AO3, making title searches impractical.

For example, linkao3(Sisyphus by esama) results in the bot linking to a translation, Сизиф (https://archiveofourown.org/works/11494326). Linking the full URL, linkffn(https://archiveofourown.org/works/1113651), results in the correct fic, but the display isn't right; it says that the author is esamaSharedosisi_rambles, which is a combination of the correct author, esama, plus the above translation author, Sharedo, and the author of a related podfic, sisi_rambles.

Handle (usually accidental) replies to bot's post

Steps to Reproduce / Scenario:

1. User A makes a post with linkffn() or similar directives, triggering the bot's (B) reply with links and summaries.
2. User C intends to reply to A but inadvertently replies to B's post.

Current behavior:

C's post is ignored by B (as far as I can tell).

Suggested behavior:

Some combination of the following:

• B detects C's reply and sends A a private message with a link to C's post and possibly its summary.
• B detects C's reply and sends C a private message warning them that they've probably replied to the wrong post.