nikosbosse / metaculus-twitter-bot Goto Github PK

The deploy.sh script needs to be triggered and the credentials stored safely

The bot currently tweets new questions up to 2 times, which is not ideal. Current issue:

• the minimum_hours is set to 48 to make sure that the question was already at least 24 hours old before we compare its past value to the one now. So in order to get labelled as 'change' the question needs to be at least 48 hours old.
• The 'new question' tweet ignores that and tweets it right away, then waits 24h because of the no-tweet-period and then tweets it again as new. Then in the next cycle it would be labelled as 'change'.

Some thoughts:

• I was thinking that maybe the requirement we really want is that a question must have x (currently 10, maybe 8?) forecasts at the first time point against which it is compared. So maybe it is fine for a question to published at t=0, it has 10 forecasts at t=10 and we tweet about a change in 24h at t=34 (or a change in 5h at t=15)?
• So we could tweet it as 'new' iff a question has a) more than 10 forecasts and b) didn't have more than 10 forecasts 24h ago? And if it has more than 10 forecasts at both time points it would be a regular change?
• Downsides: a) it could (in theory) happen that a forecast gets tweeted as new when it crosses the threshold after a month or so (could also add a time requirement). b) This would however make the code potentially more complicated.

What do you think @Scott-Huston, @edomt, @adam-binks?

We recently had an issue with the Metaculus Alert bot double posting stuff, see https://twitter.com/MetaculusAlert/status/1563949726097195008.

What I did for now was just deactivate the bot altogether because I'm currently absolutely swamped with things, but that's of course not ideal. Does anyone have time to take a quick look at what might have happened and how to fix it?
@edomt @adam-binks @Scott-Huston
Thank you very much!

One way to to this would be to implement a Kolmogorov-Smirnof-test to test whether the two distributions are the same.

For questions involving percentages, is the difference in percentage points (6% -> 12% would be 6 pp instead of 100% increase)? I think that is more intuitive, and if it's already that, perhaps the tweet can explicitly say 'pp.'

Make text easier to skim by

• adding emojis
• potentially add a screenshot / plot

One proposal:

Currently the questions to check are hard-coded. One could get a list of all open questions from the API here (https://www.metaculus.com/api2/questions/). Ideally, this should be queried using the URL to avoid a lot of unnecessary computations for the bot.

Check the questions before tweeting about them. One possible check would be when the question was created. E.g. only questions that are older than X should be taken into account.

We're currently omitting questions that are less than 48hours old. Instead, we could make a tweet that a new question is published.

Sylvain from Metaculus suggested that the bot could in the future regularly include top comments (available on the API here: https://www.metaculus.com/api2/comments/)
My thoughts on this are:

• It could post a tweet and also post an automatic reply to that tweet with the top comment
• the top comment could be defined as
  • comment with the most upvotes
  • that passes a threshold of e.g. minimum 5 upvotes
  • the comment was made in the last X hours, where x could e.g. be equal to 24h + the time delta (i.e. the 5 or 24 hours)
• We could also post a second reply with the overall top comment from that question (although less convinced about that).

@edomt @adam-binks @Scott-Huston

The bot has access to the time series of past community predictions. It could therefore check whether the previous run of the bot would have tweeted an update. If there is no significant change after the last update, one could omit the question.

Currently we are calling the Metaculus API independently for every question. In principle we should be able to get all/most relevant information from a single API call, allowing us to reduce the time the bot needs to run.

What do you think about either:

A) Opening the bot up to all Metaculus questions

B) Making a second bot which is all Metaculus questions (or all non-Ukraine questions)

For me personally, I'm most interested in updates in longtermist/x-risk questions, even though I imagine they're rarer. So it'd be great to see these on Twitter as they happen.

(E.g. motivating hypothetical example: everyone is talking about a new Deepmind paper, and then you see a tweet saying that Metaculus's AGI by 2050 question has just had +10%!)

Given that Metaculus links now have a nice auto-generated preview with a chart & more info, I think we should do away with our custom-made chart for the bot. So that:

would instead be:

What are your thoughts?

For 6 days in a row, the bot has claimed double digit percent increases in this one question while the actual prediction increased only from 31% to 35% in this time.

https://twitter.com/MetaculusAlert/status/1604718578200199173
https://twitter.com/MetaculusAlert/status/1605141407894839296
https://twitter.com/MetaculusAlert/status/1605533993025372160
https://twitter.com/MetaculusAlert/status/1605986967421861904
https://twitter.com/MetaculusAlert/status/1606439947615043585
https://twitter.com/MetaculusAlert/status/1606892905544794115

https://www.metaculus.com/questions/11243/russian-military-deaths-by-2024/?sub-question=10557