charcoal-se / halflife Goto Github PK

The internal architecture is rather messy, and should be rearchitected.

Must:

• Report results when they are established. The current architecture attempts to collect results into a dict and then separately report what's in the dict when everything is done, but this means the analysis and reporting code is always in two different places, usually far away from each other. The original rationale for this design was so that the analysis code would not need to know what it's reporting (for example, a function which queries DNS for information about an IP address doesn't and basically shouldn't know which domain name or Metasmoke post this is related to, but those details are relevant and important when reporting the result).
• Modularize ActionCable and Metasmoke APIs. There is some brief attempt at encapsulating these, but it's not very intuitive or elegant.

Should:

• Break out more of the analysis into separate routines or even separate code modules (DNS, URL analysis, etc).
• Report results as JSON instead of, or in addition to, the current ad-hoc human-readable formatting.
• Explore async (#1).
• Explore breaking out URL fetching into a separate microservice.

The restored "x/y over timespan" calculation was wrong, it counts x and y per individual feedback, not per post!

Regex queries have been disabled in Metasmoke so the client should stop attempting to make them.

Ref: https://chat.stackexchange.com/transcript/message/48992963

I created a metasmoke ticket, might be a duplicate? Charcoal-SE/metasmoke#596

The logic is not accurate, and should use a different approach.

A month ago I had to restart Halflife a number of times after the month had rolled over, and now I'm seeing the same thing again.

In brief, it seems to eat up all the disk space, and require a number of restarts before the space is properly reclaimed.

This could be a weird artifact of the Docker deployment model and/or how it works on the EC2 instance where I'm running this. Probably the deployment model should be reworked altogether.

When two or more posts are reported in rapid succession, the second and subsequent posts are sometimes missed by Halflife.

Searching for tagged/skipped shows many such incidents.

I originally thought this was a problem with Metasmoke, but in fact, it's probably a bug in Halflife. It should pick up the websocket again quickly after a message is received in order to keep on listening properly.

Moving to async processing would probably fix this. (#2 #1)

Regex search over the Metasmoke API was turned off for stability and performance reasons. (See #6.)

I have a fix in the works for domains, but URL tails (mostly useful for updating drugs brands) doesn't look like it's going to be easy.

The SO chat interface abbreviates long URLs. This is not visible in the chat log so it's not really meaningful to link to examples here, but Jake noted as much with a copy/paste here:

https://chat.stackexchange.com/transcript/65945?m=44952582#44952582

He suggests adding a line break or something.

Frequently when a post is reported, you would like to see very quickly the first results from the analysis: is the domain name blacklisted? did the URL tail match any keywords? and only then proceed to report the more-detailed analysis.

The chat interface throttles sending to one message per second. Some messages could be collected into multi-line chat messages (but then you cannot use formatting -- no bold, italics, links etc) to make reports appear quicker.

Ultimately, I'm thinking the back-end queries should be done using some sort of asynch framework so that multiple queries could be pending at the same time and you don't have to wait for the queries to execute serially before you can get the result from the one you actually care about for this particular post.

https://metasmoke.erwaysoftware.com/post/164468 contains an entire Bachelor's degree PDF on a low connection which took forever to download and caused Halflife to crash with Memory error:

https://chat.stackexchange.com/transcript/message/49448449#49448449

https://stackoverflow.com/questions/22346158/python-requests-how-to-limit-received-size-transfer-rate-and-or-total-time has some ideas for how to abort a download when it keeps going for too long.

sevenmentor.com in particular is notoriously switching IP addresses like others change underwear. It would be nice if Halflife could periodically check their IP address, and perhaps even automatically add the new address to the IP watchlist when it changes.

(My practice has been to keep the old addresses on the watch list still. If we find that this is problematic, maybe also eventually figure out a way to remove old ones.)

The bot stopped posting messages at 07:30 UTC this morning.

The last message is https://chat.stackexchange.com/transcript/message/44783748#44783748

It's not entirely clear what caused this, but Art of Code suggests switching to the v2 API.

https://chat.stackexchange.com/transcript/message/44785359#44785359

https://chat.stackexchange.com/transcript/message/61810642#61810642 shows a bit of a train wreck in the URL parsing logic.