No more results after ~ 1 week about hashtag-count

[flegfleg]

Sorry for my late answer @cstephen
Thanks for your efforts, it works for me now, too.
👍

[cstephen]

Hi @flegfleg, I'm in the process of trying to replicate this. I've had a process running for 3 days so far. I'll let you know if it's still reporting results after 1 week, but I'll keep it running indefinitely and see what I can find.

Since you needed to create new Twitter app credentials to get it working again, I'm wondering if your old credentials were blocked by Twitter. If so, I wonder what caused it. The Twitter Streaming API is supposed to keep connections open infinitely long unless something unusual happens. (See "Disconnections" on this this page.)

This module needs to do a better job of reporting disconnects and attempting to reconnect. I'll create a new issue for that.

[flegfleg]

Dear @cstephen, thanks for your reply!
I am quite new to node.js, but if you find a way to add reporting, that would be great.

[cstephen]

Just a quick update, my process has been running for 10 days now and is still getting hashtag results. @flegfleg, are your original Twitter app credentials still blocked even after a week has passed?

I've explored the underlying twit module that the hashtag-count module uses for its Twitter Streaming API connections. The twit module has some built-in safeguards to limit the number of connection attempts in a short amount of time. It will wait longer to connect or reconnect if it knows Twitter is rate limiting it.

hashtag-count benefits from these same safeguards, but I think you can still run into trouble if you stop/start your script too often or try to run multiple instances of hashtag-count simultaneously using the same set of Twitter application credentials. I've run into situations where my script has had to wait 30+ seconds to reestablish a connection. So far I have not had my application credentials blocked permanently, however.

I've started working on issue #2. The next version of hashtag-count will communicate connects/disconnects and stop outputing false 0 results while in a disconnected state.

This issue (#1) will serve as a reminder that I need to warn hashtag-count users about Twitter's rate limiting / blocking in the README.

[cstephen]

@flegfleg, a new version of hashtag-count (1.1.0) has been published that no longer records false 0 results when the Twitter Streaming API connection is inactive. It also provides three new optional callback functions connectingCb, reconnectingCb, and connectedCb that allow you to react to connect/disconnect events. You can upgrade your version by running npm update.

See the new "Connection status callbacks and a warning about rate limiting" section of the README and the updated example scripts.

Also, for what it's worth, my test process has been running continuously for 14 days without experiencing any disconnects. I'm going to keep it running for as long as I can to see if any issues come up.

I was able to force disconnect events for development by setting my twit connection timeouts unsually low (10 seconds instead of 60 seconds), but I did that in a separate process.

[flegfleg]

Thanks a lot @cstephen!
Quite busy with other projects right now, will check it out!

[cstephen]

I'm closing this issue since I've verified that a hashtag-count process can run longer than 1 week without experience any problems, and the latest version is now able to report disconnect/reconnect events.

I'm assuming the Twitter Streaming API credentials were simply blocked by Twitter in this case after some mysterious (undocumented) repeated connection threshold was crossed. I don't think we can ever be sure without some comprehensive timing experiments, but I don't think we can do that without abusing Twitter.