cstephen / hashtag-count Goto Github PK

How can I end the steam connection?

Is there a function such as this:
hc.end()

I noticed you are always appending new keys to the same {} for results. This can grow to an unruly size and pretty quickly if left alone for awhile.

May I suggest you only return the results for the single interval? Leave it up to us users to reconcile if we need to.

If anyone else is wanting a quick solve here is what I did

const intervalCb = (err, results) => {
		if (err) {
			console.log(err);
		} else {
			console.log(results);
			// delete the key in the object. It's passed by reference so we can clean it up here.
			const key = Object.keys(results)[0];
			delete results[key];
		}
	},

People using this tool to watch the output in real time (rather than storing it, for example) will prefer to see the times in their local time zone. Why not show how to convert from UTC time to their local time zone in the README and example scripts?

I'm not 100% sure if it's possible to unit test an ongoing/unlimited hashtag-count process, but I haven't done enough research into the possibilities yet. Everything else should be testable though, I think. This is important for maintaining backwards compatibility.

Hi, thanks for your script, i am using a variation of the "unlimited.js" tracking ~40 hashtags, and works fine.

But after around one week, the script returns 0 results for all the hashtags i am tracking… restarting does nothing, I have to create a new twitter app (credentials, keys) and enter that in the config in order to make it run again.

Any ideas how to solve this would be greatly appreciated.
Thanks, Flo

Although Twitter's Streaming API connections are designed to remain open indefinitely, there are several reasons a client can be disconnected:

• Twitter server restarts or configuration changes
• Internet problems on the client's side
• A client establishes too many connections using the same Twitter app credentials

See the "Disconnections" section of this page for a more complete list.

Currently, if this module gets disconnected from Twitter's Streaming API, it continues to run like nothing happened except that all hashtag counts show up as 0. This needs to be fixed. This module should:

• Report disconnection errors.
• Attempt to reconnect, report each of these attempts, and indicate how much time was spent in an unconnected state so users are aware of how much data is missing.

The following error caused my script to exit after running for ~3 weeks:

Error: Error parsing twitter reply: `89793","indices":[13,36],"media_url":"http:\/\/pbs.twimg.com\/ext_tw_video_thumb\/895412798070689793\/pu\/img\/M4Y57VvnZ40Z52rj.jpg","media_url_https":"https:\/\/pbs.twimg.com\/ext_tw_video_thumb\/895412798070689793\/pu\/img\/M4Y57VvnZ40Z52rj.jpg","url":"https:\/\/t.co\/sPQAK28OrF","display_url":"pic.twitter.com\/sPQAK28OrF","expanded_url":"https:\/\/twitter.com\/DiazKai\/status\/895412845818503170\/video\/1","type":"video","sizes":{"small":{"w":340,"h":604,"resize":"fit"},"large":{"w":720,"h":1280,"resize":"fit"},"thumb":{"w":150,"h":150,"resize":"crop"},"medium":{"w":600,"h":1067,"resize":"fit"}},"video_info":{"aspect_ratio":[9,16],"duration_millis":6167,"variants":[{"bitrate":320000,"content_type":"video\/mp4","url":"https:\/\/video.twimg.com\/ext_tw_video\/895412798070689793\/pu\/vid\/180x320\/tYenM019Ngjn0n61.mp4"},{"bitrate":832000,"content_type":"video\/mp4","url":"https:\/\/video.twimg.com\/ext_tw_video\/895412798070689793\/pu\/vid\/360x640\/j0W-x7R89z8JW0ti.mp4"},{"bitrate":2176000,"content_type":"video\/mp4","url":"https:\/\/video.twimg.com\/ext_tw_video\/895412798070689793\/pu\/vid\/720x1280\/1cdqK9nqtd9pXiNV.mp4"},{"content_type":"application\/x-mpegURL","url":"https:\/\/video.twimg.com\/ext_tw_video\/895412798070689793\/pu\/pl\/NZfk3ZWWZ_pHusHU.m3u8"}]}}]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"en"},"is_quote_status":true,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[],"urls":[{"url":"","expanded_url":null,"indices":[134,134]}],"user_mentions":[{"screen_name":"SouljaHoward","name":"\u3164\u3164\u3164*","id":323369295,"id_str":"323369295","indices":[3,16]}],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"en","timestamp_ms":"1502948583929"}`, error message `SyntaxError: Unexpected string in JSON at position 5`
    at Parser.parse (~/hashtag-count/node_modules/twit/lib/parser.js:44:28)
    at Gunzip.<anonymous> (~/hashtag-count/node_modules/twit/lib/streaming-api-connection.js:139:21)
    at emitOne (events.js:96:13)
    at Gunzip.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at Gunzip.Readable.push (_stream_readable.js:134:10)
    at Gunzip.Transform.push (_stream_transform.js:128:32)
    at Zlib.callback (zlib.js:598:14)

hashtag-count needs to be modified to handle/ignore errors like this so they don't kill the process.

With the addition of three more possible callbacks, passing all optional settings into the start() method all at once is starting to become unruly:

hc.start({
  hashtags: hashtags,               // required
  interval: interval,               // required
  history: history,                 // optional
  intervalCb: intervalCb,           // optional
  connectingCb: connectingCb,       // optional
  reconnectingCb: reconnectingCb,   // optional
  connectedCb: connectedCb,         // optional
});

It should remain possible to set them this way, but it would be even better to support setting them more directly and individually like this:

hc.intervalCb = function () {
  // Some code.
};

This also eliminates the need for identically/awkwardly named intermediate variables to store the callbacks.

I need to insert the count result into DynamoDB on every interval.
But after reading your source code, it seems the intervalCb could only be a synchronized function.
I saw you used async.until, so I wonder if it is possible to make intervalCb async?