armindoflores / ao3_api Goto Github PK

I think something like this should be the default, with a customizable time for a fic to be a WIP:

def convert_status_to_string(meta: Dict) -> None:
    if not 'status' in meta:
        return
    if meta['status']:
        meta['status'] = 'Complete'
    elif 'date_updated' in meta and meta['date_updated']:
        date = datetime.strptime(meta['date_updated'], '%Y-%m-%d')
        if (datetime.now() - date).days < 365:
            meta['status'] = 'Work in Progress'
        else:
            meta['status'] = 'Incomplete'
    else:
        meta['status'] = 'Incomplete'

While using my discord bot someone found a work that just fails to load for some reason that I haven't been able to work out. The screenshot below shows another work successfully loading, and then the one which one load right afterward.

This was the link to the work that fails: https://archiveofourown.org/works/22833073/

When I was searching for an API for AO3, I was specifically looking for something that could be used to help sort through hundreds of private bookmarks in order to build a collection of public recommendations to share with others. Unfortunately, with the current limits of this API, it looks like I can only add bookmarks to collections rather than invite works to them, meaning I'd have to manually add each work.

Any chance you plan to make/are working on something that allows me to add/invite works to collections based on workid?

So, the title is probably a bit confusing so let me just explain it. For example, if you have an AO3 fic such as https://archiveofourown.org/works/17297594 and call work.authors with that fic, it will return:
[<User [TheCrazyFriend]>, <User [Omnicyde]>, <User [Omnicyde]>]

But if you look on the website, the only author listed is TheCrazyFriend and Omnicyde is listed twice within the notes section.

This can probably be sorted by just adding a bit to where you break down the beautiful soup object.

Hope that makes sense.

The "single_chapter" argument for Search() isn't working. It's pulling works with multiple chapters.

1. Cannot get text when text not in 'p' label
   import AO3 work = AO3.Work(25693009) work.chapters[5].text
   chapters[4].text returns empty due to the 'table' label
   I am not sure why the text is not in the 'p' label. Like the pic below. chapters[5].text works well.
   

2.same code as (1) searching words : 如何迅速
I want to download the book into '.txt' type. But chapters[5].text uses ‘< br >’ instead of '\n'.
Thus when I write chapters[5].text into file, the sentences are linked tightly,not in original paragraphs.

Sorry for my poor English. Thks for your reading anyway:)

Work.comment() isn't working and throws an AuthError instead

No matter what I do, I can't get work.get_comments() to work.
Code:
from time import time import bs4 import requests import AO3 work = AO3.Work(24560008) work.load_chapters() start = time() comments = work.get_comments(1, 5) print(f"Loaded {len(comments)} comment threads in {round(time()-start, 1)} seconds\n") for comment in comments: print(f"Comment ID: {comment.comment_id}\nReplies: {len(comment.get_thread())}")
Error:
Traceback (most recent call last): File ".\ao3.py", line 8, in <module> comments = work.get_comments(1, 5) File "C:\Users\nithi\AppData\Local\Programs\Python\Python38-32\lib\site-packages\AO3\works.py", line 272, in get_comments ol = div.find("ol", {"class": "pagination actions"}) AttributeError: 'NoneType' object has no attribute 'find'

Specifically, if it comes across a fic with a blank word count (<dd class="words"></dd>) it throws a ValueError because '' can't be cast to an integer. Should be a fairly minor fix.

An example of a fic: https://archiveofourown.org/works/27776251. It appears to be some weird bug in Ao3 itself, and the fic was later reposted by the same author but has all the attributes.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\Python39\lib\functools.py", line 969, in __get__
    val = self.func(instance)
  File "C:\Program Files\Python39\lib\site-packages\AO3\session.py", line 507, in bookmarks
    n = span.split(" ")[1]
IndexError: list index out of range

This was a weird one to diagnose, but I was able to determine the issue by messing around with the URL in my browser. When logging in to my session, I was able to successfully login with one of the letters in my username being capitalized where it isn't on the archive, so I didn't notice any issues until I tried to get the number of bookmarks.

Say for instance you have the username Test_user, and you enter your login details as Test_User, this will work fine and you won't encounter issues on the site. If you then go to the bookmarks page using Test_user as the username, it shows up as you would expect, with the bookmarks tab selected in the navigation. However, if you go using Test_User, then the navigation will not have the bookmarks selected, despite showing your bookmarks.

I'm unable to install your API on repl.it. After some searching it seems that the issue might be due to an outdated python library. Is there any way you can update it?

Here is the actual error I'm receiving:

--> python3 -m poetry add ao3-api
Using version ^2.0.6 for ao3-api

Updating dependencies
Resolving dependencies...

[AssertionError]

exit status 1


Repl.it: Package operation failed.

Work.subscribe() isn't working and throws an AuthError instead

AO3 started detecting the python-requests user agent and slowing down its response time as can be seen below:

import requests

def req_python_useragent():
    requests.get("https://archiveofourown.org/works/12432?view_adult=true")

def req_normal_useragent():
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36"}
    requests.get("https://archiveofourown.org/works/12432?view_adult=true", headers=headers)

>>> print(timeit(req_python_useragent))
45.93747901916504

>>> print(timeit(req_normal_useragent))
0.6435115337371826

I checked the X-Runtime header and it's about the same for both requests.

Depending on how authors are adding images and what AO3 is doing with that, I'm running into a few works which return a KeyError because there is a blank img tag with no src defined.

(For reference the above code is from here)

Is there a way you can have chapters.get_images exclude blank img tags so other images are still retrievable?

I'm not sure if it was intended to be this way, but shouldn't the oneshot property of a work return true if it has one chapter and is considered complete? Currently, it only accounts for the former condition, but this means multichaptered fics that's only got one chapter for now are counted as oneshots as well.

Wondering why I'm not able to import the library on VSCode or any online IDE. Still a beginner so maybe it's an obvious issue I'm missing.

Can you please make a version that runs with url requests to a heroku server. I want to implement this into an iOS app and it seems like it will be tricky without plain url requests.

When using the Search function, one can call hits using a Constraint value, to refine the search by the number of hits the fics are allowed to have to be returned. However, it seems that this does not actually affect the search results, as far as I can tell from testing different permutations of the below code. The other functions that rely on Constraint values function as intended, but when searching for hits the results returned ignore the hits value while still following any other parameters given.

search = AO3.Search(hits = AO3.utils.Constraint(1700,1725))
search.update()
print(search.total_results)
for result in search.results:
  print("{} has {} hits".format(result.title, result.hits))

I'm not sure what happened, but series.work_list is returning a blank object. It was working fine a couple of weeks ago, but I was working on my bot and noticed that it's returning blank lists. I checked it with a couple of different series, and even checked it without manipulating the data.

So if is_subscribed is called for a Series object, the following error is returned:
form = self._soup.find("form", {"data-create-value": "Subscribe"}) AttributeError: 'NoneType' object has no attribute 'find'

However, its gets even weirder as it still returns the correct value (True if the user is subscribed to a series and False if they aren't).

Maybe you could shed some light on why this is.

I was trying to pull info from work.metadata and was getting no response. I took a look at the source code in the 2.0.7 release package and it looks like work.metadata and updates associated with it aren't present, even though your release notes state that it was included in the update. Just wanted to let you know.

Threads (parent comments) are loaded using one request per page and that's fine. But why do you need one request per thread to load the sub comments instead of loading them all at once and not having to do so many requests?

P.S: thanks a lot for your work, these API's have been extremely useful :)

While trying to parse some author's info, the following error will be thrown:

AttributeError                            Traceback (most recent call last)
<ipython-input-33-c6973bd07654> in <module>
     11     for result in search.results:
     12         work = get_work_data(result.id)
---> 13         get_author_data(work.authors)
     14     if DEBUG:
     15         break

<ipython-input-27-0560239b64d9> in get_author_data(authors)
     19         AttributeError: 'NoneType' object has no attribute 'find'
     20         '''
---> 21         print('Works bookmarked: {}'.format(user.bookmarks))
     22         print('Number of works: {}'.format(user.works))
     23         print('Bio: {}'.format(user.bio))

~\anaconda3\lib\functools.py in __get__(self, instance, owner)
    965                 val = cache.get(self.attrname, _NOT_FOUND)
    966                 if val is _NOT_FOUND:
--> 967                     val = self.func(instance)
    968                     try:
    969                         cache[self.attrname] = val

~\anaconda3\lib\site-packages\AO3\users.py in bookmarks(self)
    278 
    279         div = self._soup_bookmarks.find("div", {"id": "inner"})
--> 280         span = div.find("span", {"class": "current"}).getText().replace("(", "").replace(")", "")
    281         n = span.split(" ")[1]
    282         return int(self.str_format(n))

AttributeError: 'NoneType' object has no attribute 'getText'

While using the session.get_bookmarks() method, the API throws the following error:

Traceback (most recent call last):
  File "c:\Users\spaze\scriptdev\projects\ao3\bookmark-export\main.py", line 14, in <module>
    bookmarks = session.get_bookmarks()
  File "C:\Users\spaze\AppData\Roaming\Python\Python39\site-packages\AO3\session.py", line 446, in get_bookmarks
    self._load_bookmarks(page=page+1)
  File "C:\Users\spaze\AppData\Roaming\Python\Python39\site-packages\AO3\threadable.py", line 13, in new
    return func(*args, **kwargs)
  File "C:\Users\spaze\AppData\Roaming\Python\Python39\site-packages\AO3\session.py", line 478, in _load_bookmarks
    new = Work(workid, load=False)
UnboundLocalError: local variable 'workid' referenced before assignment

I am unable to say with perfect confidence that bookmarked deleted works are the cause of my problem. However, inserting a print statement in session._load_bookmarks() right after new = Work(workid, load=False) shows that quite a few of my bookmarks are loaded successfully before the error is thrown. And obviously, I do have a few bookmarks in my list that are deleted and could be the cause.

Sorry if this isn't that helpful, I've never actually opened an issue myself before.

When running the first snippet of code in the README saved as a .py file to get the WorkID. I get an error message that it cannot import from functools.

The resulting error message occurs regardless of if I run it from the ao3_api or AO3 directory.

Fresh Ubuntu 18.04 LTS in VirtualBox.

(project) ao3dude@ao3dude-VirtualBox:~/project$ python3 ~/project/ao3_api/snippet.py 
Traceback (most recent call last):
  File "/home/ao3dude/project/ao3_api/snippet.py", line 1, in <module>
    import AO3
  File "/home/ao3dude/project/ao3_api/AO3/__init__.py", line 2, in <module>
    from .comments import Comment
  File "/home/ao3dude/project/ao3_api/AO3/comments.py", line 5, in <module>
    from .users import User
  File "/home/ao3dude/project/ao3_api/AO3/users.py", line 2, in <module>
    from functools import cached_property
ImportError: cannot import name 'cached_property'

Here are the commands I used to setup the environment:

# Update the OS software and grab some Python packages. 
sudo apt update && sudo apt upgrade -y
sudo apt install python3-dev python3-pip python3-venv git

# Make a directory and start a Python virtual environment in it.
mkdir projectfolder 
python3 -m venv --system-site-packages projectfolder/
source projectfolder/bin/activate

# Move in to the folder, get some needed Python modules and clone the git.
cd projectfolder/
pip install --upgrade pip setuptools
git clone https://github.com/ArmindoFlores/ao3_api.git

I was testing out the 2.1.0 release, but I couldn't seem to have any success working with work.get_images using the code which worked with version 2.0.8. After some testing, I realized that it's only occurring with works that are only a single chapter even if they have images. Works which have images and are multiple chapters are fine.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\Python39\lib\site-packages\AO3\works.py", line 37, in __init__
    self.reload(load_chapters)
  File "C:\Program Files\Python39\lib\site-packages\AO3\threadable.py", line 13, in new
    return func(*args, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\AO3\works.py", line 81, in reload
    if "Error 404" in self._soup.find("h2", {"class", "heading"}).text:
AttributeError: 'NoneType' object has no attribute 'text'

Expected result: Trying to create a Work instance using an ID of a work that has been hidden by being added to a collection (shows up on the site as a Mystery Work) should either raise a descriptive exception about it being hidden or should cause the object to be flagged in some way that the user can detect.

Actual result: An extremly unclear error is thrown because the existing error handling expects that the page shows is actually a 404 error, and treats it as such without checking.

An example of a work that will throw this exception is 20336386, and the page can be seen at https://archiveofourown.org/works/20336386

I believe that simply raising InvalidIdError, as the class already does for a work that simply couldn't be found, with a different message would be an acceptable behavior. Additionally, it looks like a <p> block with the class "notice" is something that may be unique to this page, and could be used to detect this.

After running a search, attempting to get the workid of a work from search results page fails.

Here is the code (simple.py), normally this should print the workid of the first result of the search results page.

# Written in Python 3.8.5 64-bit, tested on Ubuntu Desktop 20.04.02 LTS.
import AO3

print('AO3.VERSION is:', AO3.VERSION)
search_results = AO3.Search(word_count=AO3.utils.Constraint(5000, 15000000),fandoms="Supernatural", language="en")
search_results.update()

workid = search_results.results[0].workid
print(workid)

Here is the output:

~$ python3 simple.py
AO3.VERSION is: 2.1.0
Traceback (most recent call last):
  File "simple.py", line 17, in <module>
    workid = search_results.results[0].workid
AttributeError: 'Work' object has no attribute 'workid'

I also tried this with an older script that I know had worked previously (AO3.VERSION: 2.0.3) which also fails with the same error message if I run this same file.
The code in this file is just ripped out a previous working file so I suspect there has been some change on the ArchiveOfOurOwn website search results that has caused this.
Given the timing I think #44 is probably affected by this.

When a work is deleted, trying to get its info causes an AttributeError (more specifically, on the line work.h4.find_all("a"))

Hey there,

do you think you could provide an endpoint that allows accessing the chapter data? As in, chapter title, summary, #words...? :) That would be grand!

Thank you.

hello,
I made myself a script to download every fic with new chapters.
it worked fine at the beginning until I had too many subscriptions and got rate limited so I added a delay to avoid the issue, but now I'm having a Connection Broken error. I don't think it's a problem with my internet connection

File "/usr/lib/python3/dist-packages/urllib3/response.py", line 425, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 755, in read_chunked
    chunk = self._handle_chunk(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 699, in _handle_chunk
    value = self._fp._safe_read(amt)
  File "/usr/lib/python3.8/http/client.py", line 611, in _safe_read
    raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(5977 bytes read, 4263 more expected)

here is the relevant code

def loadWorkMetadata(work):
        work.reload(load_chapters=False)
        sleep(DELAY)
    return None

def loadWorksMetadata(works):
    threads = []
    loadedWorks = []
    print(f"starting to check you {Fore.CYAN}{len(works)}{Fore.RESET} subsciption")
    for work in works:
        loadedWorks.append(work)
        newThread = threading.Thread(target=loadWorkMetadata, args=([work]))
        newThread.start()
        threads.append(newThread)
    for thread in threads:
        thread.join()
    return loadedWorks

Session.get_marked_for_later returns a list of Work objects. However, when those objects are initialized, the session object is not passed (c.f. session.py line 560).

The result is that calling methods of the work which require a session (e.g. an attempt to call work.reload() on a restricted work) will fail.

I believe this can be solved simply by passing session=self when initializing each work. I'll issue a PR.

Thank you for your work on this library!
Since there is a function to leave kudos, is it possible to make one for creating bookmarks?

I noticed that when using the API to make searches, I can get the information that's in the banner (tags, summary, kudos count, etc.). However, since kudos, comments, and bookmarks are only listed in the banner if they are greater than 0, they are only stored if they are greater than 0. Otherwise, they're left at None, and if you try to access the property later, it crashes when it tries to find these in the page.

Do you think it is a reasonable solution to assign the kudos, comments, and bookmarks properties to 0, in the get_work_from_banner function, if they are not found in the banner? I can't think of any other cases where they wouldn't be listed, but I'm not sure if there is a reason it's best to leave it as is, and I shouldn't assume I can get this information without loading the fics.

...

https://pypi.org/project/ao3/

This one also installs an ao3 (lowercase) package, and the way Python handles these things is that installing the two at the same time will result in a single ao3 (lowercase) package in site-packages directory, with broken namespacing and cryptic errors. I don't think anyone is using that library, unless they're sleepy like me when setting up the environment, but just FYI.

AO3.Work objects can't be pickled after being loaded because they raise a RecursionError exception.

works.py seems to be missing the Rating (Not Rated, General Audiences, Teen And Up Audiences, Mature, Explicit) property

(by the way, thank you so much for making this api!)

<p></p> tags aren't formatted properly and the string returned by Work.get_chapter_text() doesn't contain the original newlines.

When searching for a work via it's title the search results return as something I don't understand.
Searching the code doesn't reveal anything about how the results are returned or supposed to be understood.

search_result = AO3.Search(title="The Real Meaning of Idioms")
search_result.update()
print(search_result)

Returns this result in the terminal:
<AO3.search.Search object at 0x7f5ab4ea6100>

I did at first think this was due to a title collision, but have tested this will multiple different titles.

Pickled AO3 objects store the page html instead of only storing the needed attributes. Probably should make it so all attributes are calculated once reload() is called instead of relying on functools.cached_property to handle this.

This is an obscure case and very frustrating, in the latest 2.2.0 version.

If there are no bookmarks for a series, the label is not there either

<dt class="stats">Stats:</dt>
<dd class="stats">
<dl class="stats">
<dt>Words:</dt>
<dd>23,327</dd>
<dt>Works:</dt>
<dd>16</dd>
<dt>Complete:</dt>
<dd>No</dd>
</dl>
</dd>
</dl>

So when I ask for series.nbookmarks it returns this error: UnboundLocalError: local variable 'book' referenced before assignment

I tried to work around it, but my try... except Exception handling doesn't get the error very well, nor does hasattr(series, 'nbookmarks'). Could you fix this, preferably by returning 0?

Thanks

using the example in the readme, the output is:

https://archiveofourown.org/users/bothersomepotato

The Archive will be down for 2 hours of maintenance between 05:00-07:00 UTC on Thursday, October 1 (what time is that for me?). Please check @AO3_Status on Twitter for updates.

2

The HTML looks like this:

  <div class="clear"></div>

</div>

  <div class="alert announcement group" id="admin-banner">
    <blockquote class="userstuff">
      <p>The Archive <b>will be down for 2 hours</b> of maintenance between <b>05:00-07:00 UTC on Thursday, October 1</b> (<a href="https://bit.ly/2Gjf2zA" rel="nofollow">what time is that for me?</a>). Please check <a href="https://twitter.com/AO3_Status" rel="nofollow">@AO3_Status on Twitter</a> for updates.</p>
    </blockquote>
      <form action="/users/username/end_banner" accept-charset="UTF-8" data-remote="true" method="post"><input name="utf8" type="hidden" value="&#x2713;"/>
        <p class="submit">
          <input type="submit" name="commit" value="&times;" title="hide banner"/>
        </p>
</form>  </div>


<!-- END header -->

I think it can be fixed by adding this to the soup path <div class="bio module">

Currently, enabling this package's rate limiting options still doesn't prevent an HTTP 429 response.

Currently, the work is loaded with view_full_work=true. For large works, this may cause unnecessary burden on both the user and the user and AO3. There are many instances where only the metadata is required, so why fetch the whole work?

I suggest the following system:

def reload(self, metadata_only=True):
    url = f"...&view_full_work={"false" if metadata_only else "true"}"
    self._fully_loaded = not metadata_only

Then, attributes that require the entire work will call reload(metadata_only=False) if not self._fully_loaded

# first chapter only attributes
authenticity_token
authors
bookmarks
categories
characters
comments
complete
date_edited
date_published
date_updated
expected_chapters
fandoms
hits
is_subscribed
kudos
language
metadata
nchapters
oneshot
rating
relationships
restricted
series
start_notes
status
summary
tags
title
url
warnings
words

# full work attributes
get_comments
get_images
end_notes
text

# first chapter only interactions
delete_bookmark
download
download_to_file
get_comments
get_images
is_subscribed
leave_kudos
unsubscribe

# entire work interactions
comment # I'm unsure here. Where do comments on the entire work go? The last chapter?
get_comments
get_images
load_chapters

Clearly, most of the time the entire work is not necessary, hence why I would make metadata_only default to true.

The readme doesn't have work.load_chapters() before getting chapters, so following the example shows errors. I can fix the readme directly if you'd like.

PS It's working fine on Python 3.8 in Anaconda, thank you!

I have run a code snippet in a Jupyter Notebook without any issues, and it's still working. However, when I put the same code in a .py script, and try to run it, the search results field is empty after every query, no matter the page number. What am I doing wrong?

{'any_field': '', 'title': '', 'author': '', 'single_chapter': False, 'word_count': None, 'language': '', 'fandoms': 'Harry Potter - J. K. Rowling', 'rating': None, 'hits': None, 'bookmarks': None, 'comments': None, 'completion_status': None, 'page': 4, 'sort_column': '', 'sort_direction': '', 'results': [], 'pages': 14906, 'total_results': 298119}

Several files require BeautifulSoup (search.py, users.py, series.py, extra.py, session.py, comments.py, utils.py and works.py) but it's not installed or listed as a requirement by ao3_api either on GitHub or PyPI.

Should this be listed as requirement in the README or included as part of the PyPI install?

Work.leave_kudos() isn't working and throws an AuthError instead

I am getting the following error while using ao3_api on Python 2.7.17:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/rachnera/.local/lib/python2.7/site-packages/AO3/__init__.py", line 1, in <module> from . import extra, utils File "/home/rachnera/.local/lib/python2.7/site-packages/AO3/extra.py", line 23 print(f"Downloading from {url}") ^ SyntaxError: invalid syntax

I'm also getting this error with Python 3.7.5:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/rachnera/.local/lib/python3.7/site-packages/AO3/__init__.py", line 2, in <module> from .comments import Comment File "/home/rachnera/.local/lib/python3.7/site-packages/AO3/comments.py", line 5, in <module> from .users import User File "/home/rachnera/.local/lib/python3.7/site-packages/AO3/users.py", line 2, in <module> from functools import cached_property ImportError: cannot import name 'cached_property' from 'functools' (/usr/lib/python3.7/functools.py)