Download comments? about lasim

[CMahaff]

Download would be possible, but upload would not. So it would only let you read your old comments.

[ipkpjersi]

I whipped up a quick PHP script this evening for scraping user data (comments, posts, etc), it's not ideal and it could be a lot better, but it's at least something, and the pagination works nicely: https://gist.github.com/ipkpjersi/3ce754391ab8390f23d9b9f80fca3994

I'm still hoping a gui-based approach will be added, or maybe something officially through Lemmy itself, but at least this is a decent stopgap for now.

I think one of the cool things about being able to export your own comments is you could in theory create a web application for reading the comments from JSON and displaying them in a readable manner as a web page, I think that would be pretty cool.

edit: Also, it only seems to work with lemmy.ml and not instances like lemmy.one or lemmy.world? For example, this API call returns nothing: https://lemmy.one/api/v3/user/?username=ipkpjersi&limit=50

[ipkpjersi]

Good to know, thanks. Downloading is still worth it IMO.

[ipkpjersi]

If I put a bounty on this feature, would that help prioritize development of it?

Today helped me realize that this is actually an important feature to me. It seems that lemmy.one is down, and it may be gone for good since the API says it is having database issues (and if they don't have working backups it's probably just gone). If it really is gone for good, that means my comment history is gone too, which makes me realize how important this feature is in terms of having an archive.

[CMahaff]

That's very kind of you, but in all honesty probably not, simply because my time to work on this is very limited, and the more I think about it, the more I think this request might be out of scope of this tool.

I have to consider that if I add this, downloads will take a lot longer, it's more APIs I have to support, it's more data in the profile versions I have to transfer, it's more things I have to test each release, etc. All of these things make it harder to get new releases out when the Lemmy API changes, which right now, is a lot - and I think most people get more value out of timely releases than an archive feature.

That said I do think there is value to such a service, so if I have the time / energy (which I can't promise) I might fork LASIM and use the software base to make such an archiving tool. It sounds counterintuitive at first, but I think that would actually be less work overall since an archiving tool wouldn't need all the profile/uploading stuff, but would need some new APIs. If I did make such a tool it would fetch your posts, comments, and maybe saved posts, while LASIM would continue to focus on settings, subscriptions, and blocks. In conjunction, you'd basically have everything.

But again I can't promise I'll have the bandwidth since my personal life has gotten very busy, but I'll be sure to let you know if I pick it up - or if I hear anyone else has for that matter.

Could also be something to PR Lemmy about, especially since I think some kind of account export is supposed to be required to meet the GDPR?

[ipkpjersi]

That's pretty disappointing to hear that it probably won't happen, honestly, especially after it does seem like I have lost my own comment history data from lemmy.one going down.

I looked into doing it myself, because honestly if you want something done, that's the best way to do it. This, unfortunately, isn't looking much better.

I couldn't find any API endpoint for downloading my own comments from an instance. Lemmy's API documentation isn't actually API documentation, it's literally just them talking about a JavaScript/TypeScript HTTP client: https://join-lemmy.org/api/ and doesn't actually explain any of the API endpoints.

I assumed this would literally just be one endpoint for comments by user, then maybe one endpoint for posts by user, etc but it seems like maybe it's not that simple?

[CMahaff]

Yeah the documentation is pretty poor unfortunately.

I usually look at the rust API directly.

This file shows all the endpoints: https://github.com/LemmyNet/lemmy/blob/0.18.3/src/api_routes_http.rs

Then you can find all the types here: https://github.com/LemmyNet/lemmy/tree/0.18.3/crates/api_common/src

In your case, I think you want to make a GET request to <instance>/api/v3/user

You would pass a JSON object containing the fields shown in GetPersonDetails (see link above) and receive a JSON object GetPersonDetailsResponse (see link above)

But yeah, I'm not sure if you set limit in GetPersonDetails high enough if you can get them all in 1 go, or if you'll have to write a loop that pulls them down in pages until there are no more.

[CMahaff]

If your account is on lemmy.world you can probably just pass your username, but if you want to access that profile from another instance you probably need to pass [email protected].

[ipkpjersi]

Yeah but if I'm making a GET request to https://lemmy.one/api/v3/user/?username=ipkpjersi&limit=50 presumably it would just return all of the posts local for that account on that instance, but it literally returns a 404 error instead. (I have an account on each of those 3 instances I listed, with posts on all of them IIRC)

Maybe difference instances have different API versions or something? But they're all on roughly the same backend version. Maybe some of them require some sort of authentication like a token?