roachbones / discordless Goto Github PK

The exporter scripts have configuration constants defined at the top. Let's make those commandline arguments.

It seems like it could be more space efficient if a database was used to store the message data, rather than json dumps. Message data could be hashed to check if it already exists in the database, and if it does, an array of timestamps of when the message was seen can be added to. That way you get the same data fidelity, just in a more efficient form.

Hi - having a problem where I can't start the proxy server. I get this output when I run the command provided.

K:\discord backups\discordless\discordless>mitmdump -s wumpus_in_the_middle.py --listen-port=8080 --allow-hosts '^(((.+\.)?discord\.com)|((.+\.)?discordapp\.com)|((.+\.)?discord\.net)|((.+\.)?discordapp\.net)|((.+\.)?discord\.gg))$' ?discordapp\.com was unexpected at this time.

I have python 3.10.6 installed, and mitmproxy installed correctly.

EDIT: It's because '^(((.+\.)?discord\.com)|((.+\.)?discordapp\.com)|((.+\.)?discord\.net)|((.+\.)?discordapp\.net)|((.+\.)?discord\.gg))$' uses single quotes/apostrophes and not " double quotes.
Also while I'm here, mitmproxy is misspelled as mitproxy

Hi, I have setup the project correctly following the guide on my Mac, treating it as linux but using brew for MITM instead.

For some reason the dce exporter never exports anything

user@macbook-air discordless % python3.9 dcejson_exporter.py

 🧿 Initializing export 🧿 

Analyzing REST traffic.
Analyzing websocket traffic.
Collected 306 messages, 3 attachmentoids, and 41 CDN images.
Exporting DiscordChatExporter-style JSON to dcejson_exports/export_1688050158.
skipping unidentified channel 1091166109899694180. ><'
skipping unidentified channel 1108422195778175048. ><'
skipping unidentified channel 1121872920277438505. ><'
skipping unidentified channel 1010921346848542810. ><'
skipping unidentified channel 869240824142102558. ><'
skipping unidentified channel 869237617789648906. ><'
skipping unidentified channel 1090471714888102009. ><'

Exporting 0 assets... >.<'
Export saved to dcejson_exports/export_1688050158
0 assets saved to dcejson_exports/export_1688050158/assets
Finished in 0 minutes.

 ✨ All done. UwU ✨ 

The html exporter works though

user@macbook-air discordless % python3.9 html_exporter.py   
Collected 306 messages and 1 attachments from 7 channels.
Sorted 50 messages in the * channel. (channel id *)
Prepared chatlog.
Rendered chatlog.
Chatlog saved to html_exports/*/*.

Sorted 50 messages in the * channel. (channel id *)
Prepared chatlog.
Rendered chatlog.
Chatlog saved to html_exports/*/*.
...

And examining the htmls I can see the messages

Whenever I try to use mitmproxy with the Discord desktop app, it displays the error Client TLS handshake failed. The client does not trust the proxy's certificate for discord.com (OpenSSL Error([('SSL routines', '', 'sslv3 alert certificate unknown')]))

In the documentation it says to "add the certificate in Chrome", but I'm unsure what that means.

Both exporters currently store every message from every channel in memory at once. To avoid running out of memory, maybe we should use more disk reads instead.

Idea: for each channel, observe only the requests relevant to that channel. For the /messages endpoint, the URL can determine relevancy without viewing the contents, since it contains the channel ID. After exporting the channel, clear observations.

Gateways will be tougher to optimize in this way.

This will still require holding an entire channel's worth of messages in memory at once, which a user could still surely fill up their memory with, but we need to do that to export them to JSON to HTML anyway.

The icon specified by iconUrl in an embed's author isn't showing up in DCEF. Not sure why. Tried printing the message object to console in DCEF, but that option does not seem to do anything.

I might be missing some program that wasn't listed in the tutorial, but whenever I try to run discordless on windows, mitmproxy crashes.
I followed all the steps and it gives me the error AssertionError: Unexpected event type at HttpStream._handle_event: Expected Start|HttpEvent, got Reply(WebsocketStartHook(flow=<HTTPFlow in the wumpus_in_the_middle.py command.
Is there a way to fix this?