Scalability (# of messages)? about dmarchiver

[Mincka]

There is no limit in the number of messages that can be retrieved.

I know someone who has been able to retrieve a conversation with more than 80 000 tweets and 900 images. 😆

The script is building the entire conversation in memory but only of the parsed content, so I think the size is not so big for the majority of conversations. The goal was to reverse the order of the conversation at the end, to output them in a file in a chronological order.

However, I think it would be also possible to prepend the reversed content of each batch of 20 tweets in a file directly. I've already added a -raw-output switch to write the HTML content to a file after each request.

I am not sure yet of the best design to allow me to add additional output types. I was thinking about splitting the HTTP receiver and the parser completely so it would be possible to work on the parsing "offline".

[muhittin]

I know someone who has been able to retrieve a conversation with more than 80 000 tweets and 900 images. 😆

165k tweets, 2262 images, 60 videos from a conversation 😎