eloquence / freeyourstuff.cc Goto Github PK

We're duplicating code for rendering sitesets in the extension and the web service right now. Let's turn this into a library so we can use it easily in both.

Let's add DataTables' print (buttons) plugin so users can easily get a PDF or print-out of their downloads.

Some users find it helpful to be able to watch a short video, to understand what they can do with a piece of software. Let's keep it simple so we can update it in regular intervals.

Now that the extension is published regularly, there's no excuse for not using a build system.

Plugins will likely break with relatively high regularity, so testability is crucial.

Will be tricky for sites where we really do need a login. For sites where we can get content un-authenticated, we should have a set of test URLs for each, and run through those regularly.

>=1,000 answers currently result in plugin failure.

If we can't find a siteset we currently just show a blank page

The plugin sometimes grabs the HTML before the MathJax JS has done its work on the DOM node. This actually leaves in some ways a more useful result (just TeX), but we should do one thing consistently.

Kindle highlights have been suggested, since they're very hard to manage currently. Look at https://kindle.amazon.com/your_highlights to see how hard they'd be to scrape

For exploratory/showcase purposes, it's nice to see the raw HTML. Rather than flawed automated inspection, it's probably fine to have a manual flag for this for now that we can regularly set as new datasets come in.

For re-users it'll be nice to have easy access to the schema. Let's add this next to the download JSON link.

This is a mess, let's clean it up.

We accept potentially unsafe input. Unless we're certain the user is the uploader we should sanitize by default. Even then we may want to be careful.

I am not sure if this project is for liberating only public content or also private content.

I have tried few times in the past to create a way to download all my Facebook content. I want more than what I can get through their download archive approach:

• I would like to have also all my likes and comments I made
• I would like to have all comments and likes others made on my content

I have tried to use Facebook API to get my own data, but it seems it is pretty limited:

• it does not allow anymore access to messages
• it does not allow to list your own friends

The best approach I have found is to scrape from user's "activity log" all activity you made, and then you can get post IDs and also fetch the rest through Facebook API, if necessary. I think having a browser extension which scrapes "activity log" and infinity loads it until the end. We can probably go around rate limits by going slowly the first time and then remembering where we left the last time.

Now that we have an FAQ, it should be findable from the rest of the site

We're showing different timezones in different contexts. Let's fix that.

We're currently only importing the latest review, but often that references prior reviews so we should detect/import them all. Since they'll all reference the subject and have a date they should be identifiable as updates without an additional key.

There's an existing export tool that generates CSV which we can use

Checking the redirect from https://www.tripadvisor.com/MemberProfile may be better than the current fragile link parsing

Let's make sure that all pages display sensible titles.

memory backend is not designed for prod usage

Under some circumstances it takes a while for the popup to open. We should show a spinner or other loading indicator.

Initially just product reviews

That Quora issue likely will come up in other contexts as well -- let's make sure our counts in all plugins take that into account, maybe write a generic helper method for it.

In particular, we'll want to load deps like papaparse as needed.

Let's make it easier to report/debug errors since we'll likely have quite a few edge cases to deal with in the wild.

iframes currently render as broken within the extension preview of a result set. That's because Chrome doesn't support them. We should use webview instead: https://developer.chrome.com/apps/tags/webview

This can be done purely client-side in result.js -- just make sure we don't proliferate this change to the downloaded or published version.

When you're logged in, the homepage (or whatever page you see upon login -> redirection) should probably show a big "download browser extension" button.

Right now, clicking on "Download the browser extension" from e.g. https://freeyourstuff.cc/user/twitter/5715a801fa092d382cfdcd83 takes me back to the homepage, which doesn't have a link to the extension.

Before the extension is in the e.g. Chrome app store, I'd suggest linking to https://github.com/eloquence/freeyourstuff.cc#extension-setup. It threw me through a loop just now :)

Right now it's not really possible to explore uploaded content. Let's make browse/ a bit spiffier.

Let's set max-width

A couple of folks have requested support for Quora answers, let's see how hard this would be.

https://savemyscrobbles.com/about.html (by Matt Lee) makes it possible to save Last.fm scrobbles.

HTML form (above), saves your username and email address into a flat file. Once a week a cronjob grabs your scrobbles with our existing Python scripts and saves them to disk.

Is this the sort of thing one could write a plugin for?

There's really no good reason to do this client-side, and this will get us closer to having a functioning site without JS.

Let's give the user a license dropdown instead of just CC-0.

One of the reviews in https://freeyourstuff.cc/view/amazon/5716d1408708521e387a83e7 ("Sheila Shine", I gather from the text of the review) doesn't have the name of the product in the Product column -- it's blank -- and thus there's also no link to the thing being reviewed. Import error?

For users with lots of content, having to download everything from scratch every time is a chore. We should have a "Everything since last download" checkbox option if the user triggers a second download. This should be doable for most plugins.

Right now we potentially respond to events in multiple windows if the user leaves one of the result tabs open.

Retrieve button sometimes is briefly enabled when it shouldn't be, make sure we always check whether the plugin is busy doing stuff.

We can probably handle pics on Yelp without blowing up our tiny VM immediately (Instagram would be another matter). That would be a good way to get the architecture for media downloads right.

• Turn bullets of available content types into checkboxes (though some content types may depend others being downloaded first?)
• Use https://stuk.github.io/jszip/ to bundle everything? Split into multiple batches if needed?
• Investigate resilience of large file uploads to web service

.. to avoid subdomain issues.

It's not entirely obvious how to find one's data after clicking the publish button. Let's make that easier!

Has been requested. Would be nice to add to support the vegetarian/vegan community

We might preserve those down the road, but for now let's not add them to the question string.

.. so we have info beyond the username of the uploader, and can download datasets again later.

We should have a regular dump run & make it easy to set up mirrors

Add documentation for writing your own plugin

Let's see if this is doable with bundling/minification

Make sure we can use these both in Node & extension.

After going through over a thousand Quora answers (thanks), when hitting "publish" I get the error:

There was a problem publishing your data. Sorry! :-( We got the following error message:
JSON data could not be parsed, invalid syntax.
You can help us out by filing a bug.

Hitting "download" doesn't help much either, as chromes gets a NETWORK_FAILED.
Maybe an issue in one of my 7 proxies?

UPDATE: I tested the "download" feature in another network and got the same error.

For usability purposes, it would be nice to show a list of supported sites when the user opens the extension on freeyourstuff.cc (and perhaps, in general).