joonaspaakko / audible-library-extractor Goto Github PK

• Does seem to load up but it just won't play: play button on the cover
• No error messages.
• Works in desktop Firefox
• Doesn't work in desktop Chrome or Safari, but does seem to work in their mobile counterparts (ios).
• However, it does work in the extension environment.
• I'm thinking it has to do with cross-origin ...stuff. My best guess is that it's related to strict-origin-when-cross-origin
• This seems related: goldfire/howler.js#1463

I don't know why, but for some reason Audible on occasion deletes or misplaces the cover image for older books in your library. From what I've seen, these are quite often books that don't have a store page anymore or it was replaced.

The point being that the copy of the book you have in your library doesn't have a cover anymore for whatever reason and this is causing some issues in the gallery.

You do still retain access to the book itself, which makes me think that it is perhaps a by product of how they handle certain changes to old books. Maybe it's a human error, who knows.

• Gallery grid will be broken wherever there's a missing cover.
• Gallery background will flicker weirdly if a book with a missing cover finds it's way there.

Going to fix this by adding a placeholder image, more or less just like what you see in your library in Audible, if there's a missing cover image.

Audible has that thing where it quite eagerly asks you to login even if you're already logged in. Like you're logged in looking at your library and then you try to buy something and it's like "login, please". Happens in various different situations, for example just going to your account details or checking the purchase history.

🐛   Apparently it may ask to re-login when checking the wishlist too and this will cause the extraction to stop without any particularly visible error messages. A good sign that this is happening is when the little progression text counter is not there at all. This happens because it can't calculate how many books you have in your library since it can't access the wishlist page before you login again. The progress bar should also not be visible either when this issue happens.

It will actually show a few error messages in the browser's console (not in Firefox) that have to do with audible-library-extractor-content-script.js and in one of them it'll say address https://amazon.com/signin or something along those lines. This is a dead giveaway it stopped because of this exact issue.

Temporary fix / workaround:

Close the extraction by pressing the X in the top right or by refreshing the page and then log out and log back in.

If wishlist extraction still doesn't go through, it's probably unrelated and you might want to post another issue about that.

Fix?:

I haven't spent a lot of time thinking about this, but I don't think there's a lot I can do about this... other than possibly showing an actual error message that explains why it stopped and then advice the user to log out and log back in. I could potentially redirect them straight to a sign-in page?

When you first tap the cover, it shows some invisible hover only elements, and then on the second tap opens book details. Just haven't gotten around to fixing that yet...

So right now the only ways to get to the gallery page, assuming you've extracted the data, is to either use a direct URL or by going back to audible.com/library, opening extraction settings and clicking the open gallery button.

What I want to add is a new right-click context menu item to the extension icon that just throws you to the gallery page.

This isn't a big issue since you can just activate the sorting again from the menu, but it is however easy to not realize it happened, especially if you're changing filters. Either way, a bug is a bug.

I feel like it's too easy to not realize what this yellow update button does or that it's even there...

So the idea is to basically to remove the update button and make it so that after one full extraction it just does an update from then on. If the user feels like they need a full extract, they'd just remove the existing data using the trash can icon and extract again.

It looks like it basically abandons the previously extracted series entirely and keeps them for any new books. You can get the series back by doing a full extraction.

What it affects:

• Series page and its subpages
• Book details: books in series
• Most sorting and filtering

This is a bit more complex below the surface, but the gist of it the issue is that the proper way to sort series by book numbers is to include the series name as a parameter (if possible) so that the sorting will happen using the correct number.

This has already been taken into consideration in the "Books I own in the series" list.

In this picture we can see the book gets sorted based on the first number, which happens to be for the other series it belongs to:

For example this link to a publisher in audible: Simon & Schuster Audio will just do a search for "Simon".

Fix:

• encode & → %26
• and also: ? → %3F

Loving this project, thanks for open-sourcing it!

One small issue I've found is that all audiobook courses from The Great Courses are omitted from the results returned from the library extraction.

Any idea why that might be?

It may actually move on eventually, but it's not worth waiting for, because it takes such a long time, achieves absolutely nothing and you'll be done faster if you start from the beginning and exclude ISBN + Wishlist from the new extraction.

• Chrome processes it without any issues.
• There are no error messages (unless you dig real deep)
• It seems like the issue is with cross origin request failing

I've had quite a few issues that stop the extraction process, which is starting to convince me that each step should save the data they gather before moving on. Right now the data is saved at the very end, so when anything like this happens, you have to try to extract everything again, which isn't good for anyone.

Basically it's a small web app that takes book covers from your extracted data: library, wishlist, collections, categories, etc... and arranges it in a grid collage and you can save that as a .png or .jpg or as an animated wallpaper.

The extension gallery will have a button in the top right that opens the editor from any page with books in it, using whatever you see in that page as the source for the book covers, meaning you can limit the image source by filtering, searching.

This editor is actually nearly done this preview uses covers form my library.

The editor could work independently from the extension, as you can see in the link... but I have no plans to do that because to add cover images, you'd have to copy & paste cover urls one by one and that seems a bit cumbersome when there's already an extension that fetches the covers automatically.

It should only be colored in the audible library page only, but now it's just basically always on.

The idea is that the part of the extension that makes the extractor appear in your library would say "activate the icon", but instead I had a brainfart and the extension just sees a tab and goes activate icon!.

I just need to append these URL parameters to all outgoing URLs that lead to Audible to avoid unnecessary redirects:

?ipRedirectOverride=true&overrideBaseCountry=true

Why?

Users outside of the US may be redirected to what is essentially the front page of another region, the region Audible thinks they belong to unless they are logged in. Even without these parameters they can still access the original page they tried to open, but only after clicking an extra link:

bug for Audible Library Extractor v0.2.6 public release.

Over the past few weeks, I have not been able to finish an extraction. It seams that it is hanging on "fetching additional data from store pages..." explicetly on Ready Player One. Looking at the console the only things I see is the errors listed below, and they seem to happen on every call, not just this one book.

audible-library-extractor-content-script.js:2 TypeError: Cannot read properties of null (reading 'getAttribute')
    at audible-library-extractor-content-script.js:2
    at NodeList.forEach (<anonymous>)
    at a.getDataFromCarousel (audible-library-extractor-content-script.js:2)
    at audible-library-extractor-content-script.js:2
    at Object.step (audible-library-extractor-content-script.js:2)
    at audible-library-extractor-content-script.js:2

audible-library-extractor-content-script.js:2 axios caught an error TypeError: Cannot read properties of null (reading 'getAttribute')
    at audible-library-extractor-content-script.js:2
    at NodeList.forEach (<anonymous>)
    at a.getDataFromCarousel (audible-library-extractor-content-script.js:2)
    at audible-library-extractor-content-script.js:2
    at Object.step (audible-library-extractor-content-script.js:2)
    at audible-library-extractor-content-script.js:2

Image of the screen when it hangs up:

And the only other notice I see is
launch-3111109c87da.min.js:29 🚀 _satellite.notify is deprecated. Please use the `_satellite.logger` API.
But this seems to be a Audible SEO issue not one with the plug in.

My thoughts are that it is the video on that page that might be causing the issue, but I am unsure on exactly how the extrntion operates to say that for sure that the video is the issue.
Link to page as per Curl
https://www.audible.com/pd/Ready-Player-One-Audiobook/B005FRGT44

Things atempted to resolve issue on my side

• Clear Browser Cache
• Clear DNS Cache
• Use in Incognito mode
• Uninstall and Reistall
• Install in another chrome session

Atached is the console log from chrome of one atempted run.

www.audible.com-1634580026467.log

Please let me know if you need me to run anything to get you more info if you are not able to repoduce it.

• Works just fine on desktop.
• I feel like it has to be some sort of mobile specific conflict that prevents it from working.

Note to self: make sure to check other sorters as well.

Conditions:

1. Open a gallery page with book details open
2. Scroll down far enough for the next chunk of books to load (scrollbar increases)
3. Try to click any covers in the next chunk and it'll fail to open

Only seems to happen if book details were open on page load.

So basically I want to add some sort of buttons in the gallery that let you export it as a CSV file.

• Options?
  • Compatibility:
    • Goodreads
      • Mostly just a plain CSV, but all books without ISBNs are removed. I'm going to have to check if that is necessary, but my understanding is that the import will fail if there are any items without ISBNs.
    • Google sheets
      • Some special formatting like here: https://docs.google.com/spreadsheets/d/1ZyLL6ThekMHN7MMJ4uoMNgoM7VxmKolxP5ZvMvgBzVs/edit?usp=sharing
  • Export source...
    • Currently visible books
    • Library
    • Wishlist

Tested using extension v.0.2.6 — Mac: Chrome Version 91.0.4472.77 (Official Build) (x86_64)

Oddly it seems like unpacked extension saves a .csv just fine, but a packed (from Chrome web store) saves a .txt file. Firefox saves .csv just fine.

Not a big deal: All spreadsheet applications should open/import the file just with either extension.

The only issue is that your operating system likely won't recognize the correct default application for CSV files. If that is an issue for you, you can just manually rename the file from .txt to → .csv and that's it.

This has been the case for a while... I decided to make this issue so people know I'm aware.

This is what it looks like now

This is where I'll move it, I think...?

Not a big bug and easy to fix, but when I noticed it I got super confused. This second heading should only show up when you click the child category.

If I had to throw in a wild guess, the issue is probably that the content of these pages, like collection/sub-page and wishlist is generated on mounted() and when you try to navigate directly from one to the other, no mounting happens in between and so nothing changes.

You can't really navigate from wishlist to another subpage, but you can from another subpage to the wishlist since it's listed in the menu

So that you can pick and choose exactly what to save in the stand-alone gallery, without having to delete extracted data from the extraction settings through audible.com/library page.

Save options

Include:

• Library
• Collections
• Wishlist

Library includes pages Categories, Series, and Collections.

Once again we can thank my sloppy coding for this. Two different errors on that page that can cause it.

• TypeError: Cannot read property 'series' of undefined

  audible-library-extractor/src/output-page/_mixins/prepSeriesSubPage.js

  Line 25 in 5a6f298

  const seriesName = _.find( books[0].series, { asin: seriesASIN }).name;

• TypeError: Cannot read property 'bookNumbers' of undefined

  audible-library-extractor/src/output-page/_components/alePages/aleGallery/aleGridView/aleSortValues.vue

  Line 64 in 5a6f298

  const seriesNumbers = _.find(this.book.series, { asin: seriesAsin }).bookNumbers;

Basically the same thing as with library except that it wouldn't update any metadata.

• Add new books and fetch additional data from store pages for these books only
  • Right now it fetches additional data for all books no matter what
• Clear removed books

This will make wishlist sorting significantly faster, especially for larger wishlists.

The interesting thing is that it's not just there in the nav when it becomes visible at <630px, rather the view mode icon appears way later...

You can see it on this page at about <450px window width: https://joonaspaakko.github.io/my-audible-library/#/categories/

Right now I'm thinking the overall fix is to make sure it actually just stays there the whole time (on gallery pages). You don't necessarily need the spreadsheet view mode at these mobile sizes... but if someone were to give you a link that leads to the spreadsheet view, it would be nice to be able to move to the grid view without having to remove any URL params.

Sometimes books just don't have a number and that makes sorting them impossible (kind of). This leads to sorted lists that aren't always accurate.

I'm fairly sure they are manually ordered based on what the publisher wants (or whatever).

Examples:

1. Monster Hunter series, has 3 numberless books. They are definitely not in the release order, but they could be in alphabetical order. I could find many series with at least 2 books that seemed to fit this mold: numberless books are at the bottom of the list in alphabetical order. But then I found two series that seem to suggest it's completely manual ↓

2. Working for Bigfoot in the Dresden Files series doesn't have a number but is placed between book 15.5 and 16

3. Another example and perhaps the better example is Jane Yellowrock series, which has 3 books between 12 and 13 and they are definitely not in alphabetical order.

So what this means is sometimes the sorting may look like this:

Possible fix:

The only reliable way to get the correct order for series would be to scrape all series pages (some have multiple pages) and sort the books based on that.

I'm not super eager to do this because it would increase the scraping time by a lot. That said, I think having the correct order might be worth the increased scraping time. It's so irritating when the order is wrong.

Easy to fix, just going to add like 100px padding below the list element.

For some reason long titles in the Books I own in the series list break the layout on mobile. It has to do with some mobile size viewport specific CSS that allows the list items to expand the container.

I believe the issue is during the wishlist extraction it maybe can't find the URL or it's somehow processed wrong and then the gallery tries to rebuild the URL from nothing...

Would be great to pull out bookmarks and notes as well.

https://github.com/vatioz/AudibleBookmarks -- has an example.

Able to add as enhancement?

It depends on what kind of computer you are using. You could see it happening on an old laptop or perhaps you don't see it at all. I'm seeing this on slightly older mobile devices when the library exceeds about 600 books.

It also depends on the size of the current list. As in if you filter your library to let's say 200-400 books, you probably won't see the issue.

It basically lags when you do anything:

• Scrolling
• Loading elements after scrolling
• Opening book details
• Everything just kind of takes its time
• The more books you add, the laggier it gets

Proposed fix:

Currently, there's basically a kind of a pseudo lazy load that keeps track of every single book's position and then renders rest of the information when the books enter the viewport on scroll. So the solution I'm going for is to further the same idea by loading books in chunks so the original solution doesn't have to keep track of so many books at once. Note: the position tracking ends when the book is fully rendered. This solution doesn't affect the initial page load time.

• I'm not sure yet if you should be able to click the individual titles here.

  • Should the book details open right there?
  • Should it open the book in the deeper grid view with the book details open?
  • Should each category block here be just one clickable area that leads to the category page?
    • This would be the least work, but I think I'll have to go with one of the other two solutions, because showing the covers wouldn't make much sense if the next click leads you to a new page with possibly a 100 books with these 5 somewhere in the mix.

• I think in this case the View all link should lead to a page that shows all books in the subcategories in that default grid view.

  • I don't know if these different subcategories should be divided by a heading. On one hand, I see that as necessary thing, but on the other hand, it could lengthen the page a whole lot, if let's say every single subcategory only has one book and the maximum amount of covers side by side is 5.

This is the current rough layout I'm going for, but perhaps without the search field...

I've not found a single use for collections in Audible myself, but including it in the gallery might make a lot of sense. Some users might enjoy seeing their collections in the gallery themselves, but you could also use collections to make lists for your friends.

Because collections are limited to books in your library I should only need to scrape each book's ASIN in a collection and then generate the gallery by filtering the full list of books to only contain those that have a matching ASIN.

Right now only one filter below the first divider line can be active at a time and it's unnecessarily restrictive. Same goes for the static values for filters below the second divider line, so those should be range sliders.

I kinda like that I can see how many books I have in different ranges at a glance right now, but it has to make way for range sliders.

Something like this:

Interactive proof of concept on how the filters should work (Strict is what I'm going for): https://codepen.io/joonaspaakko/full/oNBgjJy

The typical thing to stop an extraction is an error that simply pumps the breaks on the extraction. In this case it seems like the amount of simultaneous http requests is too much and the page freezes up and becomes unresponsive.

You can toggle book details by clicking a cover in the grid view or a row in the spreadsheet view.

At around 754px, 762px, and 766px the end of the row is calculated wrong and the book details end up breaking the grid.

Example: https://joonaspaakko.github.io/my-audible-library/#/library?search=%255Edust%2520%257C%2520lost%2520gods&sortValues=true&sort=title&sortDir=desc

Lost gods should be first based on the sorting in URL params...

I thought I fixed this already (1932da4), but I guess that didn't do it.

This can happen in certain type of wishlists. Right now I'm thinking it's caused by at least one wishlist item with a release date or added date that is maybe missing or fetching that value fails for some reason.

Console error: Cannot read property '0' of undefined at a.fixDates

The issue happens in the fixDates() method that tries really hard to format all dates consistently as yyyy-mm-dd. I'm not entirely sure what is going on yet.

The issue happens here...

...or here

Apparently I rushed the testing a bit... Whoopsie.

I don't see this as a common use case, but since it is possible to save the stand-alone gallery this way, it should work properly.

And when I extract the zip file and open the index.html, it's missing all the sub pages in the menu:

In some cases the wishlist extraction will stop and throw this error in the browser console:

Cannot read property 'length' of null
at a.timeStringToSeconds (audible-library-extractor-content-script.js:2)

• This error comes from the extension trying to process book length, when it's not available
  • Like if you have a pre-order in the wishlist or it just doesn't have the length for some reason.
• This is only distantly related to #23. Same kind of issue, but different culprit.
• The root of the issue:

  audible-library-extractor/src/content-script/_components/_mixins/main-step/process-wishlist.js

  Line 135 in 2876a43

  if (length) book.length = vue.shortenLength( length.textContent );

• Possible fix: weeding out any inputs that don't have numbers in the shortenLength() method.
• Other thoughts: This process probably fails like nobody's business in any other language and I'm not sure there's much I can do about it, aside from fix those issues when the time comes. Turns out Japenese is the only language where they don't use some form of the word minute and that works for me so I'm planning to fix it by detecting 分 as well. ^{_{This might not be the only issue in Japanese though.}}

So far it seems like it either stops extracting at 20 titles (1 page) or in the case of my test collection of over 100 titles, it shows up empty.

Any new books that get added through the update aren't getting properly (or at all) merged with previous series related data.

This bug needs to be stomped before #47 is added.

Basically, if you go to any of the subpages and click the Wishlist button, the data source is now globally the wishlist, so when you go back to the library and try to click a book in the My books in the series list, it can't show the books in the series because it's trying to find them from the wishlist.

This is a sticky value that doesn't expire until cache is cleared or the data source is changed back to Library.

Currently you can work around this issue by opening any subpage from the menu and changing the source by clicking the "library" button:

So you when you click a book cover to open book details and continue scrolling down or up, all of a sudden the scrolling jumps up to the cover. There doesn't seem to be much logic to when it happens or at least I haven't figured that out yet. However if you scroll without momentum... as in scroll, hold for like a half a second and release, it won't jump back up.

If I remember correctly, the only thing that could make this happen is there's a method that scrolls up to the cover when the book details comp is mounted... so maybe it just keeps mounting when scrolling. Right now I pretty much have no idea why it would though and it could be something else is interfering there. It definitely doesn't seem to happen on desktop.

Categories page may load up empty every single time or every so often, depending on your library contents.

I made an assumption that book categories always come in pairs: parent category > child category but apparently that is not always the case. Could also be that the book(s) that caused the issue had both categories, but the extension failed to extract one of them.

If one book is missing the child category or they're both missing for some reason, this error Cannot read property 'name' of undefined will occur and prevent the page from loading.

The problematic code:

It's supposed to be an alphabetical sort based on the publisher, but it seems more like a random sort...

If I remember correctly, publisher sort shares the sorting function with a few other sorting methods, which means the issue might not be limited to publisher.

At some point in summer of 2020 Audible renewed the library on the website (and mobile app, but that's irrelevant). Not only did they change the layout a little, they made content changes too. One of those changes was that they removed the date added = when the book was added to your library. Totally gone from the library pages. You can still sort by date added, you just can't see the dates. So my fallback for a while was to get it from the store page where it was still available, but only for one or maybe two months until they took that away as well. The possible workaround would be to also scrape the purchases page and basically connect the dots, but I don't think it's worth it.

So as the data is fetched in the order they were added to your library, you can still sort based on date added in the library extractor's gallery.

TL;DR The extension can't fetch date added.

Image of the old and new library layouts: