Chat bot able to speak natural language and markup, prompt to complete commands, and offer localized help. Configured with schema-validated YAML, features JWT authentication with granular RBAC, and SQL persistence.
License: MIT License
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![greenkeeper[bot] avatar](https://avatars.githubusercontent.com/in/505?v=4 "greenkeeper[bot]")
![isolex[bot] avatar](https://avatars.githubusercontent.com/u/451959?v=4 "isolex[bot]")
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
Add a kubernetes controller to read data from and issue commands to a k8s cluster.
v1:
Monitoring and restarting the bot, reading and writing config and secrets, clustering, backup jobs, all kinds of fun.
It'll be great.
No k8s client yet, potentially https://github.com/godaddy/kubernetes-client
Using https://github.com/kubernetes-client/javascript directly
Services should keep a Dict<string> of labels in their metadata and use them to filter transforms, select parsers, and do other clever things.
labels to:
Service and/or BaseServiceCommandMessageControllersDates are terrible and timezones will be worse. In a feeble effort to head that off, make sure everything date-related is injected via DI, and don't use Date.now.
Date or somethingTokens need to the current date to sign and verify.
Controllers, filters, the bot, and numerous other services have ephemeral state: counters, bans, filter state, etc. Providing a signal to reset these.
Reset counters and filters for testing.
Not all commands are complete when they are parsed. The bot needs to be able to prompt for remaining arguments (or the next argument). Based on https://docs.aws.amazon.com/lex/latest/dg/how-it-works.html
v1:
Fragment (data for later Command)Command from Fragment (complete)The graph implementation is currently part of the express listener. This bloats the listener and overly couples them, along with making the graph (de)serialization methods difficult to test.
src/graph pathLex's NLP is a very important part of the bot. Make sure the parser is working, intents become nouns and verbs, slots are mapped correctly, and completion handled.
v1:
v2:
The YAML parser does not report very helpful errors when an env var or file is missing.
Parse a config file with an !env and no corresponding variable or !include and a missing/invalid path.
> make run-terminal
node /home/ssube/code/isolex//out/main-bundle.js
(node:609) ExperimentalWarning: The fs.promises API is experimental
uncaught error during main: { YAMLException: cannot resolve a node with !<!include> explicit tag at line 18, column 80:
... troller/kubernees-controller.yml
^
at generateError (/home/ssube/code/isolex/node_modules/js-yaml/lib/js-yaml/loader.js:165:1)
at throwError (/home/ssube/code/isolex/node_modules/js-yaml/lib/js-yaml/loader.js:171:1)
at composeNode (/home/ssube/code/isolex/node_modules/js-yaml/lib/js-yaml/loader.js:1393:1)
at readBlockSequence (/home/ssube/code/isolex/node_modules/js-yaml/lib/js-yaml/loader.js:928:1)
at composeNode (/home/ssube/code/isolex/node_modules/js-yaml/lib/js-yaml/loader.js:1331:1)
at readBlockMapping (/home/ssube/code/isolex/node_modules/js-yaml/lib/js-yaml/loader.js:1062:1)
at composeNode (/home/ssube/code/isolex/node_modules/js-yaml/lib/js-yaml/loader.js:1332:1)
at readBlockMapping (/home/ssube/code/isolex/node_modules/js-yaml/lib/js-yaml/loader.js:1062:1)
at composeNode (/home/ssube/code/isolex/node_modules/js-yaml/lib/js-yaml/loader.js:1332:1)
at readDocument (/home/ssube/code/isolex/node_modules/js-yaml/lib/js-yaml/loader.js:1492:1)
at loadDocuments (/home/ssube/code/isolex/node_modules/js-yaml/lib/js-yaml/loader.js:1548:1)
at load (/home/ssube/code/isolex/node_modules/js-yaml/lib/js-yaml/loader.js:1569:1)
at safeLoad (/home/ssube/code/isolex/node_modules/js-yaml/lib/js-yaml/loader.js:1591:1)
at readConfig (/home/ssube/code/isolex/src/config/index.ts:63:20)
name: 'YAMLException',
reason: 'cannot resolve a node with !<!include> explicit tag',
> make run-terminal
node /home/ssube/code/isolex//out/main-bundle.js
(node:609) ExperimentalWarning: The fs.promises API is experimental
uncaught error during main: !include file did not exist
Both types have resolve methods that check whether the variable or file exists. If those return false, the parser throws the above.
!env!includeUsers belong to Groups, which have Roles, which are granted permissions (or scopes). The AuthController should be able to compile the list of applicable permissions for a particular User and check to make sure a list of permissions are all present.
The k8s controllers should require permissions to scale or update resources.
The auth controller should require permissions to edit users, issue tokens, etc.
Need to resolve the full set of roles that apply to a user and compile their permissions in a semi-performant way. Maybe look at TypeORM caching (redis?).
Use shiro-trie for permission tests.
🚨 You need to enable Continuous Integration on Greenkeeper branches of this repository. 🚨
To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.
Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.
If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.
Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please click the 'fix repo' button on account.greenkeeper.io.
Add an authentication and authorization controller.
v1:
Some commands and controllers, like k8s and gitlab, should be access-controlled.
Will require new entities for role, token, and user. No utils for permissions or tries.
Users should be attached to Roles, which are granted Scopes.
Permissions should use a Shiro-like trie structure, with foo:bar indicating each scope and following a noun:verb:args convention.
Listeners should have a way to associate a user ID/tag/snowflake with a session, in place of cookies or a token header, so users remain logged into the bot from Discord/Slack until their token expires.
!!pick this or that could return any of [this, or, that]. That's not usually what you want.
Checklist to the PickHandlerAdd a handler that uses the database to store a counter.
User rep, message stars, etc.
It would be pretty cool if reactions could be used as rep/star triggers.
In order to implement conditional transforms, select anything by label, or implement rule-based filters, we need matchers and rules.
Take an example of a pod that uses pod affinity:
apiVersion: v1
kind: Pod
metadata:
name: with-pod-affinity
spec:
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: security
operator: In
values:
- S1
topologyKey: failure-domain.beta.kubernetes.io/zone
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: security
operator: In
values:
- S2
topologyKey: kubernetes.io/hostnameThis provides a weighted set of rules to match data (in this case, labels). Revising that slightly with isolex terminology:
metadata:
kind: template-transform
name: with-pod-affinity
scope: foo
data:
matches:
- key: security
operator: In
values:
- S1string matchesregex matchesevery operatorany operatornegate flagTotal rabbit hole. Implement operators as needed.
Should be a utility class and some operators.
The bot should print its version when it starts up.
v1:
v2:
Logs.
Requires changes to the webpack config to read variables from the makefile.
Add a listener for https://slack.com/
v1:
Listener for Slack with:
v2:
Necessary for work use.
The RTM client used to receive messages cannot search and may not be able to send all of the messages supported by the bot. Look at using the web client as well.
Daemons traditionally reload their configuration and reload when they receive a SIGHUP signal. Support this.
Easy reloads when testing.
Managed/sidecar reloads in controlled pod.
Do services all handle stopping and starting over correctly? Will they be destroyed and reconnect (most likely)?
The learn controller should take the command name from user input, but for security purposes, should filter commands before saving.
v1:
Prevent learned commands for a privileged controller.
The bot does many things and there should be a way to keep track of them. Since the bot can expose an HTTP API, that would be a great place to put Prometheus metrics. Do that.
Monitoring the bot, making sure it hasn't crashed, sweet graphs, etc.
Clustering will be interesting.
The prom-client registry will need to be injected into anything which might report metrics.
Need a library for metrics: https://github.com/siimon/prom-client
Users do not and should not have passwords. Instead, upon signing up, an initial token is created with the session:create grant (and nothing else). This login token is used to log in, containing the uid and name. Login tokens should expire after a long time (30-90 days) and be treated like passwords. Users should be told in no uncertain terms to keep the login token in their password manager.
A "password reset" means deleting all existing tokens for a user, including their login token, and immediately (transactionally, even) issuing a new login token. This replaces all existing tokens and can be used to re-issue others.
Implementing passwords opens up a million security concerns. Without passwords, most of them don't exist.
Security for tokens relies largely on the secrecy of the secretOrKey used to sign them. This is taken from config, meaning it can be an !env variable or !included from a mounted kind: Secret.
Ensure users cannot get stuck without a login token.
Add anonymous roles (roles for an anonymous user) to the session controller. Any context without a user should be given these roles and their grants. Default to session:create and user:create, most likely.
In some controller, implement the sign up flow (create user, create token, reply with token) and reset flow (delete tokens, create token, reply with token). The sign in flow is already in session ctrl. Group these three and write some tests.
Github conveniently uses threaded issues and PRs, matching the bot's message neatly. It should be possible to set up a request-and-webhook based listener.
v2:
Using the bot to manage issue and PR workflows.
Needs #68 and a Github client.
The auth controller is handling too many different nouns and has become too large. It's setting off a few different lint warnings:
Having a consistent build and test process (or better, command) is a necessary part of every project. Having a reliable CI process and automatic packaging is also quite useful. Make that happen.
.npmignore is correctNobody can use it without releases, not even me.
Add support for apps/v1 resources to the k8s controller.
coreScale applications up and down, get their status.
The k8s controller will be many and varied, probably needing some common helpers (at least to set up the client and load context).
k8s client is already installed.
The bot file/class has been flagged by codeclimate as being too long, multiple methods as being too complex, and overall poor quality. This is absolutely a god class with a service locator mixed in. Fix it!
<Foo, {}> types when creating themreceivestartServicesBot itself is too long by a few lines, split parts outResources follow somewhat standard forms, but mix and match standard fields with configuration. This leads to complex types (some impossible to enforce) and endless reserved names. Kubernetes fixed this quite cleanly:
metadata:
apiVersion: v1
kind: Pod
name: foo
labels:
bar: a
bin: b
data:
# actual bodyBreaking out the metadata will allow easy definition of service definitions, better registration, and eventually tagging. It should also ease nesting and linking.
This will be a large refactor and needs to:
add Transform services
refactor Command (these are commands, not objects)
name to nountype to verbstandardize some junk:
Config vs Data namingemit config block (partial Command)msg/msgs vs message/messagesdocument naming conventions
figure out:
Context (any service that interacted with it?)The AuthController should be able to issue, save, and (in)validate tokens.
API access via the express listener.
Need permissions in place first.
Need verbs for (in)validate.
Token entity exists, fields map to JWT claims. Issue JWTs with permissions as scopes and validate.
The learn controller currently takes the command name as a run parameter. That requires anyone running the command to know what noun they are targeting, which is often asking too much.
Running a command without knowing the bot's config.
Breaking change for the learn controller (not much use yet, probably thanks to this issue).
The same way emit: always has CommandOptions to create a new command, intervals and durations should be standardized to support both cron and zeit (10s, 3 minutes, etc) expressions.
Each service instance should provide some structured documentation, fetched with the help verb, that can be used to build command line and web pages.
v1:
help verbhelp field
This isn't golang; documentation matters.
There are a few entities with a Map property, which typeorm does not handle correctly through the simple-json type.
Currently this is handled with a protected shadowStr: string property and a pair of methods:
@AfterLoad()
public syncMap() {
this.data = new Map(JSON.parse(this.dataStr));
this.labels = new Map(JSON.parse(this.labelStr));
}
@BeforeInsert()
@BeforeUpdate()
public syncStr() {
this.dataStr = JSON.stringify(Array.from(this.data));
this.labelStr = JSON.stringify(Array.from(this.labels));
}These do not scale well to multiple fields. Turn them into helpers, base classes, or something.
To support efficient calls to remote services, add a gRPC handler.
Safely executing code requires a strong sandbox. Running a gRPC server in that sandbox provides primarily-one-way communication. Since any language might be running the code and server, the method must be widely compatible.
afaik gRPC generates interfaces and shell classes from the protobuf layouts, which would have to be integrated into the make/webpack process.
The gRPC library has a number of native components, but was added to package.json a while ago.
Add a parser that can handle CLI arguments: --foo --bar=3. Convert to a Map<string, Array<string>> suitable for cmds and msgs.
Parser classCLI args are a well-known way of providing named text options, perfect for this use case. Unlike the split parser, args can handle named arguments and array values.
Need to find a good CLI library.
Not all include paths are absolute. Relative paths should use the existing config lookup logic and priority.
!include path without leading / should use config lookup logicLoading the database a relative directory, keeping the reference config portable.
Reactions should appear in the listener's stream as messages from the user, with the reaction emoji as the body. The message should otherwise be normal.
This should allow reactions (or any use of the emoji) to trigger a normal command for the bot.
v1:
v2:
Send increment/decrement commands to the counter handler.
Need to make sure emojis mid-sentence don't trigger rep changes.
Requires #10
Parsers almost always create a Command, but some need the noun/verb configured and some take dynamic values. Some even switch depending on their config. Writer a helper to encapsulate that logic.
emit stanzaEvery parser does this or should.
🚨 You need to enable Continuous Integration on all branches of this repository. 🚨
To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.
Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.
If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.
Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper App’s white list on Github. You'll find this list on your repo or organization’s settings page, under Installed GitHub Apps.
The various entity create(data) static methods are leading to a lot of duplication. Unlike reply, emit, and some of the other helpers, they take the full FooData object. The Data and Options for each entity (unlike services) could be merged, meaning they're not actually Data and Options to begin with (most likely just Options). To merge those, mark context and the like as optional (they are, esp in the GraphQL schema). That should allow the create methods to be removed.
The Bot is currently acting as a service locator and manager for itself and every other service. This should be implemented in a DI module that tracks (starts and stops) services on behalf of the bot. This module will have some helpers to create/get services within the bot's managed scope, much like the prometheus registry for metrics.
createService and getService methodsgetService (along with id)The bot is getting too big (#37).
The bot should be able to emit commands on lifecycle events.
The bot should announce when it updates.
None
Add git info to the webpack bundle, to use when announcing start.
When shutting down the bot via signal (SIGINT or SIGTERM) it may throw an error. It should shut down cleanly, closing all resources in a controlled manner.
Error [ERR_UNHANDLED_ERROR]: Unhandled error. ([object Object])
at Client.emit (events.js:171:17)
at Client.EventEmitter.emit (domain.js:442:20)
at WebSocketConnection.onError (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/src/client/websocket/WebSocketConnection.js:374:1)
at WebSocket.onError (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/node_modules/ws/lib/event-target.js:128:1)
at WebSocket.emit (events.js:182:13)
at WebSocket.EventEmitter.emit (domain.js:442:20)
at _receiver.cleanup (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/node_modules/ws/lib/websocket.js:211:1)
at Receiver.cleanup (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/node_modules/ws/lib/receiver.js:557:1)
at WebSocket.finalize (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/node_modules/ws/lib/websocket.js:206:1)
at TLSSocket.emit (events.js:182:13)
at TLSSocket.EventEmitter.emit (domain.js:442:20)
at emitErrorNT (internal/streams/destroy.js:82:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
at process._tickCallback (internal/process/next_tick.js:63:19)
Clean shutdown logs.
The Discord listener and/or other services are not being cleaned up properly.
Write docs for a bunch of common use cases. Include examples, refer to config, and everything necessary to run the example by pasting.
Readme:
Getting Started:
Auth:
Code:
github-pull-request:getgithub-pull-request:updateChat:
Help:
People should be able to set the bot up, log in, and do a few things without ever typing an original command (pasting only). That entire process, from cloning to reactions, should exist in a single doc. Too many projects omit that guide.
The database stores triggers and some other useful features, but there is no good way to update it (yet).
Schema changes require migrations or losing the database, which is bad.
The bot itself doesn't have connection info until it has loaded config. It would be possible to run migrations at that point, but should be optional (for clustered bots, testing, etc).
TypeORM has a system for migrations, but I'm not sure that will work with the build.
The express (http) listener needs to support sessions and attach them to context. This will mostly use the existing session provider class (via auth controller), but HTTP differs from existing chat listeners because users do not remain connected. HTTP has ways to fix that: headers and cookies (a header).
v1:
Webhooks should be able to authenticate.
No support for tokens yet (besides the entity).
Need to pass context to the graph methods.
🚨 You need to enable Continuous Integration on Greenkeeper branches of this repository. 🚨
To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.
Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.
If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.
Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please click the 'fix repo' button on account.greenkeeper.io.
🚨 You need to enable Continuous Integration on Greenkeeper branches of this repository. 🚨
To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.
Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.
If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.
Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please click the 'fix repo' button on account.greenkeeper.io.
Gitlab conveniently uses threaded issues and MRs, matching the bot's message neatly. It should be possible to set up a request-and-webhook based listener.
Using the bot to manage issue and MR workflows.
Requires #68 and a Gitlab client.
The bot should have an HTTP/S API for webhooks, debugging, and all sorts of other reasons.
Webhooks.
Binding to all of the data will take some changes.
No way to feed messages (replies) from a request back to the response, unless the listener can link the two method calls (receiveMessage and sendMessage).
Need to install express and graphql.
A reaction with an standard emoji (not a custom discord emoji) that is not preset in the emoji library will crash the bot.
/build/src/git.apextoaster.com/ssube/isolex/src/listener/DiscordListener.ts:170
msg.body = emoji.find(reaction.emoji.toString()).key;
^
TypeError: Cannot read property 'key' of undefined
at DiscordListener.convertReaction (/build/src/git.apextoaster.com/ssube/isolex/src/listener/DiscordListener.ts:170:55)
at Client.client.on (/build/src/git.apextoaster.com/ssube/isolex/src/listener/DiscordListener.ts:44:25)
at Client.emit (events.js:182:13)
at Client.EventEmitter.emit (domain.js:442:20)
at MessageReactionAdd.handle (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/src/client/actions/MessageReactionAdd.js:24:1)
at MessageReactionAddHandler.handle (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/src/client/websocket/packets/handlers/MessageReactionAdd.js:7:1)
at WebSocketPacketManager.handle (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/src/client/websocket/packets/WebSocketPacketManager.js:103:1)
at WebSocketConnection.onPacket (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/src/client/websocket/WebSocketConnection.js:333:1)
at WebSocketConnection.onMessage (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/src/client/websocket/WebSocketConnection.js:296:1)
at WebSocket.onMessage (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/node_modules/ws/lib/event-target.js:120:1)
at WebSocket.emit (events.js:182:13)
at WebSocket.EventEmitter.emit (domain.js:442:20)
at Receiver._receiver.onmessage (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/node_modules/ws/lib/websocket.js:137:1)
at Receiver.dataMessage (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/node_modules/ws/lib/receiver.js:409:1)
at Receiver.getData (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/node_modules/ws/lib/receiver.js:347:1)
at Receiver.startLoop (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/node_modules/ws/lib/receiver.js:143:1)
at Receiver.add (/build/src/git.apextoaster.com/ssube/isolex/node_modules/discord.js/node_modules/ws/lib/receiver.js:117:1)
at TLSSocket.emit (events.js:182:13)
at TLSSocket.EventEmitter.emit (domain.js:442:20)
at addChunk (_stream_readable.js:277:12)
at readableAddChunk (_stream_readable.js:262:11)
at TLSSocket.Readable.push (_stream_readable.js:217:10)
The bot should continue running but skip the reaction or use a default emoji/message.
This is caused by the convertMessage and convertReaction calls in the DiscordListener throwing while outside of the async error handler.
Most of the services assume the config will be present and valid. Usually this leads to nice, clean something is undefined errors (with mediocre stack traces), but properly validating the config and getting useful errors would be awful nice.
Good errors are good.
Getting validation snippets from each service without instantiating them will be interesting.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.