triggermesh / tmctl

TriggerMesh Shaker CLI to create, manage and debug event-driven integration apps.

The TriggerMesh Cloud Native Integration Platform consists of a set of APIs which allows you to build event-driven applications. Implemented as a set of Kubernetes CRDs and a Kubernetes controller, it gives you a way to declaratively define your event sources and event targets, in addition to potential actions needed in your applications: content-based event filtering, event splitting, event transformation and event processing via functions.

Getting Started

• Guides
• Documentation

Installation

To install TriggerMesh, follow the installation instructions.

TL;DR

Register TriggerMesh APIs by deploying the Custom Resources Definitions:

kubectl apply -f https://github.com/triggermesh/triggermesh/releases/latest/download/triggermesh-crds.yaml

Deploy the platform:

kubectl apply -f https://github.com/triggermesh/triggermesh/releases/latest/download/triggermesh.yaml

Namespaced installation

By default, TriggerMesh works with resources across all namespaces in the Kubernetes cluster. This requires permissions provided by ClusterRoles and ClusterRoleBindings. If cluster-wide permissions are not desired, TriggerMesh can be configured to run in a single namespace. To get the namespace-scoped installation, first, deploy the CRDs:

kubectl apply -f https://github.com/triggermesh/triggermesh/releases/latest/download/triggermesh-crds.yaml

And then, using kubectl's "kustomization" feature, install customized manifests:

kubectl apply -k config/kustomization/namespaced/

Controllers created with this command won't reconcile resources outside the triggermesh namespace, ClusterRoleBindings will have the namespace scope.

Contributing

Please refer to our guidelines for contributors.

Commercial Support

TriggerMesh Inc. offers commercial support for the TriggerMesh platform. Email us at [email protected] to get more details.

License

This software is licensed under the Apache License, Version 2.0.

Additionally, the End User License Agreement included in the EULA.pdf file applies to compiled executables and container images released by TriggerMesh Inc.

tmctl's Issues

Reject commands with bad command line arguments

I was trying
tmcli create trigger --target foo-httptarget --eventTypes my.type --headers "Content-type":"application/json"

and didn't realise I was creating a trigger instead of target (target being my intention), hence the headers parameter was invalid.

Had the CLI told me "headers is not a parameter for a Trigger", I would have quickly noticed my mistake.

Print adapter error logs on initialization

Sockeye Implementation

the docs mention the use of sockeye but deployment is left for the user to figure out.. this seems like it will cause a bit of friction to me.

Propose that we expose a way for users to create and then reference a sockeye instance?

Add examples in the `create` help

$ tmcli create
create TriggerMesh objects

Usage:
   create <resource> [flags]
   create [command]

Available Commands:
  broker         TriggerMesh broker
  source         TriggerMesh source
  target         TriggerMesh target
  transformation TriggerMesh transformation
  trigger        TriggerMesh trigger

Test if requestbin can be used with HTTP target

Panic on awss3source creation

Default `version` output`

Consider creating a command to print the version of the tool

$ tmcli version
Error: unknown command "version" for ""
Run ' --help' for usage.
2022/10/05 09:04:14 unknown command "version" for ""

Switch to another broker when deleting a broker

If I delete a broker I am "on", I expect to be set back to another broker.

$ tmcli delete --broker timur
2022/10/13 09:40:35 Deleting "timur" broker
(⎈ |prod:sebgoa)MacBook-Air:tmcli sebgoa$ tmcli brokers
foobar
sebgoa
(⎈ |prod:sebgoa)MacBook-Air:tmcli sebgoa$ tmcli delete --broker sebgoa
2022/10/13 09:40:46 Deleting "sebgoa" broker
(⎈ |prod:sebgoa)MacBook-Air:tmcli sebgoa$ 
(⎈ |prod:sebgoa)MacBook-Air:tmcli sebgoa$ 
(⎈ |prod:sebgoa)MacBook-Air:tmcli sebgoa$ tmcli brokers
foobar
(⎈ |prod:sebgoa)MacBook-Air:tmcli sebgoa$ tmcli describe
Error: cannot parse manifest: open /Users/sebgoa/.triggermesh/cli/timur/manifest.yaml: no such file or directory
Usage:

Sync local sockets on restart

Components Spec completion assistance

Add network connectivity fail over

When fetching CRD if network has a hiccup you throw an error

$ tmcli list
Error: cannot fetch latest CRD: cannot detect latest release tag: Get "https://api.github.com/repos/triggermesh/triggermesh/releases/latest": dial tcp: lookup api.github.com: no such host
Usage:
   list [flags]

Flags:
  -h, --help   help for list

Global Flags:
      --config string    Config home dir (default "/Users/sebgoa/.triggermesh/cli")
      --context string   Context (default "default")
      --debug            Enable debug output
      --version string   TriggerMesh components version (default "latest")

2022/10/05 11:05:50 cannot fetch latest CRD: cannot detect latest release tag: Get "https://api.github.com/repos/triggermesh/triggermesh/releases/latest": dial tcp: lookup api.github.com: no such host

Implement "watch" functionality

CRD not found in wrong path

$ tmcli create target
2022/10/05 09:43:25 list sources: yaml: unmarshal errors:
  line 1: cannot unmarshal !!str `Not Found` into crd.CRD

Panic when creating KafkaSource

Listing full output below.

jonathanmichaux@Jonathans-MacBook-Pro tmcli % tmcli create source kafka --bootstrapServers 127.0.0.1:54974 --topics mytopic --groupID testgroupID
2022/10/05 22:19:09 Updating manifest
2022/10/05 22:19:09 Starting container
panic: cannot deep copy []string

goroutine 1 [running]:
k8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue({0x29d81a0?, 0xc000c23458?})
	/Users/jonathanmichaux/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/converter.go:639 +0x273
k8s.io/apimachinery/pkg/runtime.DeepCopyJSONValue({0x2b17ec0?, 0xc00056d080?})
	/Users/jonathanmichaux/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/converter.go:623 +0x345
k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.SetNestedField(0x2b18c40?, {0x2b17ec0?, 0xc00056d080?}, {0xc001319818, 0x1, 0x1})
	/Users/jonathanmichaux/go/pkg/mod/k8s.io/[email protected]/pkg/apis/meta/v1/unstructured/helpers.go:209 +0x3d
github.com/triggermesh/tmcli/pkg/kubernetes.CreateUnstructured({0xc0002e4cc0?, 0x0?}, {0xc0002e4cd0, 0xf}, {0xc00059c827, 0x3}, {0xc0001ac400?, 0xc0003f2501?}, 0xc001187ba8?)
	/Users/jonathanmichaux/Documents/TriggerMesh/git/tmcli/pkg/kubernetes/object.go:92 +0x5f8
github.com/triggermesh/tmcli/pkg/triggermesh/source.(*Source).AsUnstructured(0x3366560?)
	/Users/jonathanmichaux/Documents/TriggerMesh/git/tmcli/pkg/triggermesh/source/source.go:52 +0x3f
github.com/triggermesh/tmcli/pkg/triggermesh/source.(*Source).AsContainer(0xc00048b730)
	/Users/jonathanmichaux/Documents/TriggerMesh/git/tmcli/pkg/triggermesh/source/source.go:60 +0x33
github.com/triggermesh/tmcli/pkg/triggermesh.Start({0x3388d30, 0xc0001aa008}, {0x336a5c0, 0xc00048b730}, 0x1)
	/Users/jonathanmichaux/Documents/TriggerMesh/git/tmcli/pkg/triggermesh/triggermesh.go:54 +0xa3
github.com/triggermesh/tmcli/cmd/create.(*CreateOptions).source(0xc0004c86c0, {0x0, 0x0}, {0x209061adc, 0x5}, {0xc00048b5f0, 0x6, 0x6})
	/Users/jonathanmichaux/Documents/TriggerMesh/git/tmcli/cmd/create/source.go:85 +0x439
github.com/triggermesh/tmcli/cmd/create.(*CreateOptions).NewSourceCmd.func1(0x1?, {0xc00048b5e0, 0x7, 0x7})
	/Users/jonathanmichaux/Documents/TriggerMesh/git/tmcli/cmd/create/source.go:57 +0x188
github.com/spf13/cobra.(*Command).execute(0xc0003f6500, {0xc00048b5e0, 0x7, 0x7})
	/Users/jonathanmichaux/go/pkg/mod/github.com/spf13/[email protected]/command.go:872 +0x694
github.com/spf13/cobra.(*Command).ExecuteC(0xc0001b5680)
	/Users/jonathanmichaux/go/pkg/mod/github.com/spf13/[email protected]/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/jonathanmichaux/go/pkg/mod/github.com/spf13/[email protected]/command.go:918
main.main()
	/Users/jonathanmichaux/Documents/TriggerMesh/git/tmcli/main.go:27 +0x1e

Components secrets

Replace plain values with secret references

Switch broker with `broker` instead of `context`

Consider switching broker with

tmcli config set broker foo

instead of

tmcli config set context foo

Help for Trigger

Add help

$ tmcli create trigger --help
2022/10/05 09:50:39 "--target <name>" argument is required

finalize name

Shall we switch to tm or tmctl ?

Check ability to create multiple triggers

Add delivery configuration support

Review deletion mechanics for "context"

Review the deletion of the context and/or entire broker

$ tmcli delete --broker bar
2022/10/05 09:27:05 Deleting "bar"
2022/10/05 09:27:06 Stopping container: Error: No such container: 

Caps on description of commands

right now there are some lowecase

  create      create TriggerMesh objects
  delete      Delete components

Format listing of targets

Consider writing categories P1, P2 etc...

$ tmcli create target
Available targets:
---
awssqs
jira
awslambda
googlecloudpubsub
logz
googlecloudworkflows
cloudevents
infra
logzmetrics
oracle
alibabaoss
awss3
azuresentinel
elasticsearch
googlecloudstorage
googlesheet
ibmmq
awssns
zendesk
awseventbridge
kafka
confluent
hasura
http
sendgrid
slack
azureeventhubs
googlecloudfirestore
splunk
awsdynamodb
uipath
tekton
twilio
awscomprehend
awskinesis
datadog
salesforce

Add `delete` command

The remove command should stop the container and remove the object from the manifest. In the case of the delete <broker>, the broker configuration directory should be completely removed from the disk.

`dump` command with compatibility mode

Document Available Commands and Parameter(s)

Create documentation surrounding the available commands, sub-commands, and parameter(s).

Installation "quickie" for linux distro

no apt, no yum

something simple...maybe a silly curl commands

curl https://triggermesh.io/get-tmctl

Prompt before deletion

I want to check the syntax of the delete command and I ended up deleting my broker

$ tmcli delete 
2022/10/05 09:14:23 Deleting "bar"

Maybe prompt for ack before deletion

Help for target does not mention `sources`

$ tmcli create target cloudevents --endpoint https://sockeye-sebgoa.k.triggermesh.io/ --source foobar-awssqssource
2022/10/13 09:50:34 Updating manifest
2022/10/13 09:50:34 creating object: spec processing: property "source" does not exist, available values are: adapterOverrides, credentials, endpoint

Improve commands output

1. Add the "next steps" tip
2. create <type> command should print the list of available kinds
3. Some commands could have an option to graphically represent the events flow

Support multiple target/source/trigger

tmcli list vs tmcli get-brokers

tmcli list seems a bit cryptic

Maybe consider using tmcli get-brokers or tmcli brokers

Put the cli in brew

so that we can do

brew install tmcli

Add events transformation functionality

Support passing HTTP headers to HTTPTarget

e.g. tmcli create target httptarget --endpoint https://eodzpxvbbuf31hr.m.pipedream.net --method POST --headers Content-type:application/json

Add CI automation

prep everything to be able to release

Provide a quickstart bundle

Consider adding a simple bundle that provides a quickstart: webhook to sockeye

Something without credentials

tmcli quickstart

Docs/title on default output

We need some type of banner documentation in the output

$ tmcli
Usage:
   [flags]
   [command]

Available Commands:

Describe cloudevents target endpoint

We should be able to see the endpoint for a given cloudevent target.

Right now we do not

Target                       Kind                  Status
foobar-cloudeventstarget     cloudeventstarget     online(http://localhost:63535)

Add `send-event` command

Command should send CE to Broker endpoint

:Feature Request: Deploy Arbitrary Container Images as Services

I would like to be able to deploy arbitrary container images as services and define the respective environment variables and port mappings. I.E.

tmcli create service foo --env=["bob":"isnice","tom":"isnot"] --port=[8080:8080,50001:5001]

OR

tmcli create service --config=./config.json

WHERE config.json ==

{
name : foo,
env:
 [
  bob: isnice,
  tom: isnot,
],
port:
  [
   8080:8080,
   50001:5000
   ]
 }

Allow creating a catch-all Trigger

If I try to create a Trigger with no filter, I get this output:

jonathanmichaux@Jonathans-MacBook-Pro tmcli % tmcli create trigger --target foo-httptarget
2022/10/05 09:40:01 "--source " or "--eventTypes " argument is required

It would be great to create a catch-all trigger with no filter on source or eventType, that just forwards all broker events.

Add a dump to docker-compose

we should be able to dump to a docker-compose file

tmcli dump --docker-compose

Top priority sources to support:

• KafkaSource
• WebhookSource
• CloudEventsSource
• AWSSQSSource
• AWSKinesisSource
• AzureEventHubSource
• HTTPPollerSource
• GoogleCloudPubSubSource

Top priority targets to support:

• KafkaTarget
• HTTPTarget
• CloudEventsTarget
• AWSKinesisTarget
• AWSLambdaTarget
• AWSS3Target
• AWSSQSTarget
• AWSEventBridgeTarget
• AzureEventHubsTarget
• GoogleCloudPubSubTarget
• SlackTarget

Be specific that we delete by NAME

and we don't need to specify the kinb of the components

Proposal for Transformations in the CLI

This is a proposal for how Transformation could fit into the CLI, in a way that simplifies usage by the end user by not having to handle routing to and from the Transformation. It's inspired by EventBridge in that the Transformation is associated with a Target.

A Transformation is associated with exactly one Target, and a Target has at most one Transformation. The Transformation must transform the event before it gets sent to the external consumer (no implication here on the actual event flow).

In the CLI, after creating a Target, you can add a transformation to that Target:

tmcli create target cloudevents --name mycetarget --endpoint https://sockeye-tzununbekov.dev.triggermesh.io/
tmcli create trigger --name mytrigger --source aws-s3 --target mycetarget
tmcli create transformation --target mycetarget --from-file bumblebee.txt

Functions would also be used in the same way

tmcli create target cloudevents --name mycetarget --endpoint https://sockeye-tzununbekov.dev.triggermesh.io/
tmcli create trigger --name mytrigger --source aws-s3 --target mycetarget
tmcli create function --target mycetarget --from-file myfunction.py

Under the hood

When mapped to the Kubernetes object model, this produces a multi-legged flow:

1. Trigger from Broker to Transformation
2. Transformation responds to Broker
3. Trigger from Broker to the CloudEvents Target

Alternatively it could also be a simple linear chain: Trigger from Broker to Transformation => Sink from Transformation to Target

Implications/thoughts

Later we could expand to add transformations to sources. One use case for this could be to improve the metadata of an incoming event in order to simplify routing and processing further down. For example, if I'm getting generic events of type aws.sqs.message from an SQS source, but I know that there are several message types in there and want to extract the type and update the ce type header.

tmcli create transformation --source mysqssource --from-file bumblebee.txt

Alternative designs

• (currently in CLI) Transformation is a sink for a Trigger, that responds back to the Broker, and whose output event needs to be routed to another Target using a Trigger
• Transformation associated to a Trigger instead of a Target

Verify adapter images

A few of TriggerMesh components use adapter images from other components, e.g. AWS EventBridge Source is using AWS SQS Source image. Need to handle that somehow.

help for sub sub commands

Make sure we get the help

$ tmcli create source
2022/10/05 09:35:46 list sources: yaml: unmarshal errors:
  line 1: cannot unmarshal !!str `Not Found` into crd.CRD

Verify auto-completion script

And add documentation on how to set it up.

Import Trigger structures from Brokers project

YAML manifest corruption bug

If YAML manifest is being overwritten with data that has fewer lines of text, old data leftovers corrupt the manifest.