pharaun / karmator Goto Github PK

This breaks the graph web-view

Before the bot is usable on some servers it needs SSL and server auth so...

Make sure that the new ircbot replacement has support for ssl/tls upfront.

We want to have threading and concurrency in the ircbot to handle slow-handlers and so forth, however the primary issue would then be handlers that depends on a particular order of irc messages to work properly.

Should implement:

1. some sort of pre-filter that slaps on something like an auto-inc integer to track "total ordering"
2. have a pre-handler or option for ensuring that an handler can only ingest messages in a specific order
3. The main gotcha is how to expose this feature and how to make sure that the handlers can all be multi-threaded yet only process messages in a specific order for specific ones (can probably just block those order-dependent ones and make sure they process it one by one)

add in karma fountain/event detection, ie when it exceeds some per minute threshold for some specific value.

We need to eventually have a concept of nested configuration/modules/state.

Global

• per server
  • per channel
  • privmsg

Now the configuration like modules enabled, state, and other info is done in a hierchary order, ie we look up state and modules first at privmsg/channel level and if
nothing matches we then look at the per server then finally the global scope.

Now on the modules, that can be done at per tier but to support channels that removes support we may want to make it into a "list of overrides" ie you define modules at a
global level, if nothing is defined at channel or server level we use global level, otherwise we use the "local" level such as server or channel level

Now for now let's just focus on a functional... Global or server level configuration and iterate from there.

One of the biggest unresolved configuration question is how to handle "privileged" modules such as ones that react to join messages or so to make the bot join a channel.

If this was standalone we could do it trivially but if we want to have per channel configuration....?

Could make it be that if you want to customize the channel config you need to define it otherwise the defaults will be the server level configuration...

Give up on everything being plugins and instead look into privileged hooks

IE things that mostly acts on some of the protocol such as privmsg makes sense as plugins, others that needs privileged access can probably be hooks.

Could make it something easy like.

withServer foo /bar -> do
addHook channelJoin (/baz -> derp)

Or so to add a bunch of hooks on various stages and provide some default hooks like auto join a specified list of channels, memorize the channels invited, etc...

15:53:24 < a> !karma c c++
15:53:25 < karmator> a: c, 0 (0++/0--/0+-); c++, -4 (3++/7--/1+-)
15:53:29 < a> c++
15:53:30 < a> c++--
15:53:31 < a> !karma c c++
15:53:31 < karmator> a: c, 0 (0++/0--/0+-); c++, -5 (3++/8--/1+-)
15:53:37 < b> !karma c
15:53:37 < karmator> b: c, 0 (0++/0--/0+-)
15:53:41 < c> [c]++
15:53:42 < a> !karma c
15:53:42 < karmator> a: c, 0 (0++/0--/0+-)
15:54:01 < c> !karma "c"
15:54:01 < karmator> c: c, 0 (0++/0--/0+-)
15:55:09 < a> !karma "!karma c c"
15:55:09 < karmator> a: !karma c c, 10 (10++/0--/0+-)

IE:

chrome-- chrome--

Is something we should look into spam-reducing but we also should support

chrome-- firefox++

since its different topic being up/downvoted.

Surrounding context as of why an event got upvoted...

Probably can just relay on external irc loggers if they exist for the channel, provide adapter/interfaces for this, otherwise punt.

Once the bot is functional, i suspect the next few steps in making it more "HA" is to:

1. Add in support for dumping the live-state to persistent store of some form then do the exec-socket swap trick (got a proof of concept) to swap in the new code and reload the state.
2. Look into learning some more about how HA works with heartbeats so that when one IRCbot goes down the other will automatically take over and synchronize the state between both bot.
3. Staggered restart with one ircbot updating then the other one swapping.

We are in a position where it should be possible to completely port the current Karmator stack on over to this bot-core which will allow us to test how reliable/sold it is.

This is an good milestone for v3 then we can iterate on all of the rest of the features, let's work on this next.

This release will probably lose the web-api for the time being...

IE:

If someone upvote X karma, their upvote for X karma is converted into a downvote. And that person is then also downvoted, also a message should be announced.

Basically right now the code is just lower casing the text and that's usually ok, but we need to properly reduce the unicode expression and also casefold for the database storage.

http://shapecatcher.com/unicode/info/9548
http://shapecatcher.com/unicode/info/10746
http://shapecatcher.com/unicode/info/10786
http://shapecatcher.com/unicode/info/65291

Should also dig through this site to find more :)

There are several flood/ban bugs that needs to be implemented, and i don't feel like dealing with python code (i do it all the time at work) so I need to port the irc codebase into haskell.

There are several candidates. So far the prime candidates are:
http://hackage.haskell.org/package/ircbot

which depends on

http://hackage.haskell.org/package/irc

which in its 0.6.0.0 version on github turned out to be the fastest irc parser beating out all of the other parser (it was switched to attoparsec and bytestring). Still need to port ircbot over to use this new version then see what its performance is and how usable it is.

If i'm not a fan of its api I can probably look into simpleirc or a couple others and all else failing, implement my own based off the irc parsing library.

Basically implement a hook for processing/filtering all input and output from the bot so that we can do broad-scale filters such as:

1. if too long of a message, split cross lines
2. if too long -> pastebin

And for ingestion stuff such as:

1. slapping an auto-inc id on all inbound message for enforcing total order for plugins that cares about message-ordering.

found a channel with 200 irc bots named _bot so we need a postfix match so that we can filter these out.

Irc bot 3

• treat irc like a server
• take advantage of http routing concept
• http middleware

Anyway

• a module has a route defined
  This is matches -> commands
  Matches can -> nested routes.

Matches/routes can be defined as a recursive tree like so:

Data route = route match (either handle or route) | routeList [route]

So the key concept is to decouple command/handle from route and allow nested routes. This enables us to define a module with common shared state
and then provide a route to handle all of these common commands. Then in master route we just neat this module route.

This is also a powerful concept, we can build a route tree that matches on server, on irc channels, on nicknames, then assign a handle which is
really more nested routes.

We no longer need to have custom plugin on a per server or per channel basis, it's built into the routing tree.

Now a second concept, not all commands need to return a result, so we should always execute all commands so that the commands that don't return a
result can update and modify their shares state.

Third concept, we need to look into how http midwares work because they're going to have a subset of routes share a common state, so valuable to
look into how they work.

Fourth concept, we should funnel all server connection into a queue to be handled by the handler.

The handler then dispatch a thread or so with each new message to handle so that it can handle many in parallel.

Fifth concept, we may want to have hook, or I am thinking, events, for important server events such as invites to a new channel, kick from a.
Channel, server disconnection, etc... Stuff that needs privileged access, for acting on it actively we will need hooks, but for passive action the
command/plugin can listen to the event queue and act upon a event or not.

Sixth concept - midware, look into it

Seventh - output filters, this is to like line break, limit amount of output to irc, rate limit messages, and so on, (anything that cares about
output)

Reimplement the web-server part of karmator.

In interest of getting this release out reasonably soon I'm going to focus on just the karmator part in IRC then dealing with the rest later (mainly the web-server portion)

<karmator> plucas, most positive: foo (1673); bar (870); baz (784). most negative: asdf (-729); fdsa (-111); plucas (-86). your rank is 632 of 635 in positivity.

It says I am the third most negative, but then says my positivity rank is 632nd of 635. Shouldn't I be 633rd?

Since there is wide range of auth support, should look into providing hooks for integrating custom auth options.

1. Authping
2. server password
3. sasl
4. certfp
5. any other new types of auth, perhaps NickServ, channel password, etc...

We have a irc nick cleaner for the nickname itself, but not for the nick in the message.

< _foo>: _bar++

this properly cleans up the _foo into foo but not _bar.

This is to prevent evasion of bans and abuse unless you get completely new ip/nick but then if you flood it fast enough you'll get auto-banned.

Should be somewhat straight forward to have regex searches over the karma values.

However there's a few attack vectors to be aware of such as:

1. exponential backtracking regex
2. cpu/memory leaks
3. fact that the backing store is an sqlite database means we'll need to bring the values in memory for searching.

Need a blacklist of channels to never join.

Basically we want to have karma be case-folded properly so that it can deal with unicode rules and avoid some of the weirder lowering case edge issues.

Issues:

1. There are going to be different case variants such as foo, FOO and we will want to have a way to pick one for display.
2. Don't want to allow infinite variants because then an attacker could just enumerate through all variants to flood the database (not that they can't already with karma values)
3. SQLite support

So basically what we want to do here is some form of karma voting so for ex if someone upvotes A twice and upvotes a once, the A karma value will be our display value but the score will be the sum of A and a

So basically we'll want a form of voting for the winner. There's several ways to vote such as:

1. DNC/RNC
2. Simple Majority

Downfall:

1. Need a way to bound the unique number of aliases say 5 to 10
2. Need to have more complicated query to find the winning alias

SQL:
Priority queue in SQL. Give the casefold representation a BTREE index, then max heapify, insert breaking ties around 1 in favor of the new element, repeat. This leads to the desirable characteristic that it becomes stable once the lowest element has >1 vote.

Tooling:
Some form/manner of admin commands for removing/collapsing aliases and start afresh.

Schema:
Can probably do something like this schema

1. Master votable table which holds the case-folded version with all of the up/down/side votes
2. An alias table that holds the various aliases for each of the votables.
3. etc...

Should add a config opt to have a list of supported channels to autojoin on startup.

Externalize some of the basic server and bot config into env-var or a config file so that I can provide a pre-built binary for the basic "karmator" bot.

We have a nickname cleaner for standardizing up/downvotes but we should also still record the actual nickname for tracking down "troublesome" stuff.

Using !karma to check your own rank also means highlights for the top 3 and bottom 3 ranked people. Instead, if !karma only shows the top 3/bottom 3, and !karma user shows your karma breakdown and your rank, you can see your rank and breakdown at the same time and not bother other users.

Some one flooded the karma bot in a channel and got 3800 karma within hours, implement a flooding -> ban algo.

Specifically messagePump & messageVacuum

15:24:47 < karmator> foobar: C, 23 (28++/5--/6+-); C++, 0 (0++/0--/0+-)
15:24:56 < pberens> !karma c++
15:24:56 < karmator> pberens: c++, 0 (2++/2--/1+-)
15:25:08 < pberens> !karma c++ c
15:25:09 < karmator> pberens: c++, 0 (2++/2--/1+-); c, 0 (0++/0--/0+-)

foo bar -- someone
!karma "foo bar"
name: foo bar, -1 (0++/1--/0+-)

this should not be happening, fix this.

Would be awesome to have an megahal or something compat with megahal out of the box as a sample plugin.

Should enchance the queue to also serve up events to the plugins so that we can handle other events such as new server, connections, disconnections, and so on.

Secondly should look into figuring out how to grant each server a unique identifier for usage in routing.

Need to have a upgrade path, and design a system that can handle nickname normalization plus unicode normalization.

Need to fix up the case-lookup for nickname and karma, also need to do unicode case-folding for the actual sql lookup

• karma_parser_endpoint = endpoints.ProcessEndpoint(reactor, './karma-parser', ['./karma-parser'])
• karma_parser_endpoint = endpoints.ProcessEndpoint(reactor, './karma-parser', ['./karma-parser'], env={'LANG': 'en_US.UTF-8'})

^ is a terrible hack but it makes it work, should make this look up the system's LANG envvar and load that in.

Basically right now we have no clean way of identifying where a message came from, so have the server emit some sort of identifier that we can then use for routing. Probably Data.Unique, but i'm not sure yet.

1. Primtive nickname defuzzer (Nick_1, nick|away, etc) (DONE)
   a. More advanced stateful tracking of nickname changes via hostmask/etc
   b. Registeration (for nickname defuzzer)
   c. Nickname defuzzer in the karma expression itself.
2. Simple Karma parsing (Multiple karma, simple postinc/dec/sides) (DONE)
3. Primitive Karma defuzzer (food ++, food++) (SEMI)
   a. Gives more trouble than its worth in the simple case, broken for c++, foobar --lol-gnu-options
   b. Need a more intelligent system
4. Nick name in the karma (Nick: win++) -> (win++)
5. Explicit support for braced karma expression such as "foo bar"++ -> foo bar++ for ex
   a. How to deal with natural braces, and nested braces and karma within the braces
   b. Ignore the karam within the braces
   c. Ignore nested braces
6. Intelligent Karma reduction
   a. foo+++++++ -> foo++
   b. Careful of #3.a - c++++ -> "c++"++ and c++--++ -> "c++--"++
   c. Take care of dealing with "foobar --gnu-args" -> No karma up/down
   d. One way of handling karma collapse is -> "c++--" -> "c++, Downvote", but "c++----" -> "c++, Downvote", which would allow for "a++--" types of karma, but would fall apart if we have specific cases where we want/don't want it so may need to have an escape hatch
7. Precinc/dec (++foo) (REJECTED)
   a. gnu style command args
   b. So rarely used
8. Lowercase the outbound karma and nicknames (SEMI-DONE)
   a. Issues on the karmator side, need to track down
   b. casefolding and unicode NFC - Data.Text.ICU
9. Add support for blacklisting certain nick/hosts from interacting with the karmabot
10. Disallow self upvote/downvote/sidevote (SEMI-DONE)
11. Identify if submitted by adium (ie adium does -- -> — just downvote adium as such (adium--)) (FEATURE REQUEST)
12. Karma cap
    a. Per channel
    b. Per nick
    c. Floating/self calculating based on historical data/volume
13. Karma events
    a. Enough upvote -> |/ Karma Fountain |/
    b. Enough downvote -> Karma Drain

We want to support various bot-related events such as auth, connection, disconnection, etc... seems like it would be useful to support a subset of that via plugins so we should extend the message queue that is pumped to the router stuff with events.

There will always be some subset of functionality that probably requires a direct hook or so, but let's try to keep this to a reasonable amount.

Main> establishTLS testConfig testPersistent
** Exception: HandshakeFailed (Error_Packet "partial packet: expecting 4672 bytes, got: 4037")

Main> establishTLS testConfig testPersistent
** Exception: HandshakeFailed (Error_Protocol ("certificate rejected: FQDN do not match this certificate",True,CertificateUnknown))

< pberens> a-b--
< pberens> !karma "a-b"
< karmator> pberens: a-b, 0 (0++/0--/0+-)
< pberens> a-b--
< pberens> !karma "a-b"
< karmator> pberens: a-b, 0 (0++/0--/0+-)
< pberens> [a-b]--
< karmator> pberens: a-b, -1 (0++/1--/0+-)

Come up with some game modes for karma such as:

1. Socialist mode - redistribute/control the karmafountain
2. Karma prestige
3. Award portion of your karma to other things.

Some IRCD will apparently require you to do an "auth ping" in which it pings and you need to pong back before it will allow you to join channels and be registered.

Should probably break up the handshake portion of the code to be able to handle the auth ping step and then delay channel joins till the bot is registered.

The parsec parser does a pretty good job, but I think it may in the end be better to break it up into at least a tokenizer step so that we can reason over tokens and do a 2 stage parsing process.

Most importantly this can be subject to spam so should try to make it as succulent as possible and maybe even rate-limit it.

Have a way to query the system with different reporting requests such as:

1. time decay of x% a day so we can see what the most highly voted item was recently
2. normalized karma over some time-frame says days/week/months
3. Record when karma fountain events happens.

Sometimes people accidentally change karma for something they didn't mean to - usually because of typos, sometimes because they simply spelled someone's name wrong. This can lead to an abundance of singular items with +1/-1 (or even 0) karma polluting the database. It would be nice to have some commands, issuable only by authorized users, to remove such erroneous entries.