Coder Social home page Coder Social logo

tyktechnologies / tyk-operator Goto Github PK

View Code? Open in Web Editor NEW
190.0 37.0 35.0 19.53 MB

Tyk Operator for Kubernetes

Home Page: https://tyk.io

License: Mozilla Public License 2.0

Dockerfile 0.17% Makefile 1.57% Go 96.08% Gherkin 1.83% Smarty 0.35%
tyk-operator k8s operator-sdk api-gateway

tyk-operator's Introduction

Tyk Operator

Go Report Card

Tyk Gateway is a modern, ultra-performant, purpose-built, and open source API Gateway.

Tyk Operator brings Full Lifecycle API Management capabilities to Kubernetes. You can configure Ingress, APIs, Security Policies, Authentication, Authorization, Mediation by using GitOps best practices with Custom Resources and Kubernetes-native primitives.

API Management with Tyk Operator


Introduction | Documentation | Learning with Videos | Quickstart Examples | IDE Integration | Community


Introduction

What can you do with Tyk Operator?

Tyk Operator can configure Tyk Gateway as a drop-in replacement for standard Kubernetes Ingress. You can manage your API definitions and security policies with it. It also works with the Classic Portal so you can manage your Classic Portal declaratively.

Custom Tyk Objects are available as CRDs and documentation for each of these custom resources are available here.

Tyk Licensing

Tyk Operator and Tyk Gateway are both 100% Open Source. Tyk Operator will operate on a single gateway.

Feel free to reach to our commercial team (or your account manager for existing customers) if you need advice about architecture, licensing, or just to discuss your requirements for runnning in HA, scaling across clusters, nodes & namespaces.

What benefits Tyk Operator has?

You can get the benefits of GitOps with declarative API configurations:

  • Security and Compliance: All changes must go through peer review through pull requests. The configurations are versioned in your version control system and approved by your API Product Owner and Platform team.

  • Kubernetes-Native Developer Experience: API Developers enjoy a smoother Continuous Integration process as they can develop, test, and deploy the microservices and API configurations together using familiar development toolings and pipeline.

  • Reliability: With declarative API configurations, you have a single source of truth to recover after any system failures, reducing the meantime to recovery from hours to minutes.

Documentation

Read more from our Official doc site.

Learn about our CRDs:

Learning with Videos

Quickstart Examples

HTTP Proxy

apiVersion: tyk.tyk.io/v1alpha1
kind: ApiDefinition
metadata:
  name: httpbin
spec:
  name: httpbin
  do_not_track: false
  use_keyless: true
  protocol: http
  active: true
  proxy:
    target_url: http://httpbin.org
    listen_path: /httpbin
    strip_listen_path: true

TCP Proxy

apiVersion: tyk.tyk.io/v1alpha1
kind: ApiDefinition
metadata:
  name: redis-tcp
spec:
  name: redis-tcp
  active: true
  protocol: tcp
  listen_port: 6380
  proxy:
    target_url: tcp://localhost:6379

GraphQL Proxy

apiVersion: tyk.tyk.io/v1alpha1
kind: ApiDefinition
metadata:
  name: trevorblades
spec:
  name: trevorblades
  use_keyless: true
  protocol: http
  active: true
  proxy:
    target_url: https://countries.trevorblades.com
    listen_path: /trevorblades
    strip_listen_path: true
  graphql:
    enabled: true
    execution_mode: proxyOnly
    schema: |
      directive @cacheControl(maxAge: Int, scope: CacheControlScope) on FIELD_DEFINITION | OBJECT | INTERFACE

      enum CacheControlScope {
        PUBLIC
        PRIVATE
      }

      type Continent {
        code: ID!
        name: String!
        countries: [Country!]!
      }

      input ContinentFilterInput {
        code: StringQueryOperatorInput
      }

      type Country {
        code: ID!
        name: String!
        native: String!
        phone: String!
        continent: Continent!
        capital: String
        currency: String
        languages: [Language!]!
        emoji: String!
        emojiU: String!
        states: [State!]!
      }

      input CountryFilterInput {
        code: StringQueryOperatorInput
        currency: StringQueryOperatorInput
        continent: StringQueryOperatorInput
      }

      type Language {
        code: ID!
        name: String
        native: String
        rtl: Boolean!
      }

      input LanguageFilterInput {
        code: StringQueryOperatorInput
      }

      type Query {
        continents(filter: ContinentFilterInput): [Continent!]!
        continent(code: ID!): Continent
        countries(filter: CountryFilterInput): [Country!]!
        country(code: ID!): Country
        languages(filter: LanguageFilterInput): [Language!]!
        language(code: ID!): Language
      }

      type State {
        code: String
        name: String!
        country: Country!
      }

      input StringQueryOperatorInput {
        eq: String
        ne: String
        in: [String]
        nin: [String]
        regex: String
        glob: String
      }

      """The `Upload` scalar type represents a file upload."""
      scalar Upload
    playground:
      enabled: true
      path: /playground

Universal Data Graph - Stitching REST with GraphQL

apiVersion: tyk.tyk.io/v1alpha1
kind: ApiDefinition
metadata:
  name: udg
spec:
  name: Universal Data Graph Example
  use_keyless: true
  protocol: http
  active: true
  proxy:
    target_url: ""
    listen_path: /udg
    strip_listen_path: true
  graphql:
    enabled: true
    execution_mode: executionEngine
    schema: |
      type Country {
        name: String
        code: String
        restCountry: RestCountry
      }

      type Query {
        countries: [Country]
      }

      type RestCountry {
        altSpellings: [String]
        subregion: String
        population: String
      }
    type_field_configurations:
      - type_name: Query
        field_name: countries
        mapping:
          disabled: false
          path: countries
        data_source:
          kind: GraphQLDataSource
          data_source_config:
            url: "https://countries.trevorblades.com"
            method: POST
            status_code_type_name_mappings: []
      - type_name: Country
        field_name: restCountry
        mapping:
          disabled: true
          path: ""
        data_source:
          kind: HTTPJSONDataSource
          data_source_config:
            url: "https://restcountries.com/v2/alpha/{{ .object.code }}"
            method: GET
            default_type_name: RestCountry
            status_code_type_name_mappings:
              - status_code: 200
    playground:
      enabled: true
      path: /playground

IDE Integration

API developers may add K8s extensions to popular IDEs to enjoy auto-completion while editing Tyk CRD YAML files. Here's the detail steps.

VS Code

Watch video tutorial here.

Steps
  1. Go to the following link: https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools
  2. Click on Install. This will prompt you to open Visual Studios.
  3. Click Open Visual Studios at the subsequent prompt. This will open VS Code and take you to the Extensions' section.
  4. Click Install in the Kubernetes extension page.

Note: The extension should take effect immediately. In case it doesn't, simply restart VS Code.

GoLand

Steps
  1. Open Plugins settings following official GoLand documentation https://www.jetbrains.com/help/go/managing-plugins.html
  2. Install Kubernetes plugin (https://plugins.jetbrains.com/plugin/10485-kubernetes)
  3. Open GoLand Preferences as described here,
  4. Go to Languages & Frameworks > Kubernetes
  5. Click Add URLs and add https://raw.githubusercontent.com/TykTechnologies/tyk-operator/{version_tag}/helm/crds/crds.yaml,
    1. For example, if you would like to use CRDs of v0.9.0, replace {version_tag} with v0.9.0 and add https://raw.githubusercontent.com/TykTechnologies/tyk-operator/v0.9.0/helm/crds/crds.yaml
    2. Please add CRDs of master for latest CRDs, as follows https://raw.githubusercontent.com/TykTechnologies/tyk-operator/master/helm/crds/crds.yaml goland-support
  6. Apply and save changes.

Community

Tyk Operator is under active development.

We are building the operator to enable you to build and ship your APIs faster and more safely.

If you find any defects, please raise an issue. We welcome code contributions as well.

If you require any features that we have not yet implemented, please take your time to create a GitHub issue detailing your use case so that we may prioritise accordingly.

For larger and more in-depth feature requests, please consider starting an RFC thread in the Discussions area.

tyk-operator's People

Contributors

alephnull avatar amm55 avatar andrei-tyk avatar asoorm avatar buger avatar buraksekili avatar buraksezer avatar caroltyk avatar cherrymu avatar dependabot[bot] avatar ermirizio avatar excieve avatar firaasss avatar from-nibly avatar gernest avatar jesse0michael avatar jlucktay avatar joshblakeley avatar kofoworola avatar komalsukhani avatar olamilekan000 avatar patriziobruno avatar rdcwaldrop1 avatar rewsmith avatar sedkis avatar singhpr avatar sonjachevre avatar sredxny avatar talex-de avatar zalbiraw avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tyk-operator's Issues

makefile is currently tightly coupled to local kind cluster development

We now have helpers to improve developer workflow such as:

make boot-pro IMG=tykio/tyk-operator:test.

This is great, however it makes it difficult to use against a minikube cluster for local development, or a real K8s cluster in the cloud when testing Ingress for example.

We need to remove the hard dependency on kind load docker-image and at least support minikube for local development too.

.PHONY: install-operator-helm
install-operator-helm: cross-build-image manifests helm
	@echo "===> installing operator with helmr"
	kind load docker-image ${IMG} --name=${CLUSTER_NAME}
	helm install ci ./helm --values ./ci/helm_values.yaml -n tyk-operator-system --wait

feat: Continuous delivery github actions

Feature: Automatically build and publish latest & tag version bump versions based on SemVer
  In order for tyk-operator users to track latest or pin to a specific version of the operator
  As an operator developer
  When I merge to master
    I need to automatically tag the git repo as `latest` & bump the version
    And I need to publish a changelog and add that to the release notes in GitHub
    And the latest and versioned tag need to be pushed to DockerHub
  Scenario: Merge a pull request tagging
    Given I have a pull request
    When i merge to master
    Then github actions should automatically tag the master as latest
      And github actions should automatically bump the minor version `X.MINOR.X`
  Scenario: Merge a pull request with commit "#patch"
    Given I have a pull request
      And there is a commit message containing "#patch"
    When i merge to master
    Then github actions should automatically tag the master as latest
      And github actions should automatically bump the patch version "X.X.PATCH"
  Scenario: Merge a pull request with commit "#major"
    Given I have a pull request
      And there is a commit message containing "#major"
    When i merge to master
    Then github actions should automatically tag the master as latest
      And github actions should automatically bump the patch version "MAJOR.X.X"

--- DONE, But needs validation

  Scenario: Auto publish release notes
    Given I have a pull request
    When a release is tagged (latest or semver)
    Then A changelog will be generated from all the commits between this and the previous "latest" tag
      And this changelog will be published as release notes against the tag
  Scenario: Build and Publish to DockerHub
    Given I have a github repository
    When a release is tagged (latest or semver)
    Then github actions should build a docker image and push it to dockerhub
      And the docker image that has been pushed should have 2 tags; latest and the new github semver tag

[TT-3697] Explicitly specify context/profile when manipulating local sandbox Kube cluster(s)

The Makefile and local dev instructions and other bits look like they are dealing with local sandbox cluster(s) via tools like kubectl and kind and helm.

I hit a snag with an oversight like this on another project just recently, where an unrelated and unintended cluster became the target of one of that project's helm install calls.

I think some flags should be set to specify a context/cluster name, to prevent overrunning into other contexts/clusters that the user may already have set up and make sure that these commands are executing against the tyk-operator sandbox cluster.

Flags like so:

  • kubectl --context="..."
  • kind create cluster --name="..."
  • helm --kube-context="..."

From memory, the order of operations would start with kind create cluster --name="..." and then this name value would be the context name to feed forward into kubectl, helm, et al.

Test environment

As part of every PR, tests need to be run to test multiple combinations of resources:

ie

  • api def with token
  • api def secured with policy
  • api def secured with policy, deleted, and re-added

Ideally, the tests should run against multiple versions of the Dashboard + Gateway, but for starters, latest is fine

Todo:

  • Integration Tests
  • Unit Tests

CRUD a Tyk Organization decleratively

When spinning up a new environment, or for disaster recovery, or replication of dev/staging/prod, it is desirable to bootstrap a Tyk organisation.

{
    "_id" : ObjectId("5d67b96d767e02015ea84a6f"),
    "owner_name" : "asoorm",
    "owner_slug" : "asoorm",
    "cname_enabled" : true,
    "cname" : "portal.ahmet:3000",
    "apis" : [],
    "sso_enabled" : false,
    "developer_quota" : 0,
    "developer_count" : 1,
    "event_options" : {
        "hashed_key_event" : {
            "webhook" : "",
            "email" : "",
            "redis" : true
        }
    },
    "hybrid_enabled" : true,
    "ui" : {
        "languages" : {},
        "hide_help" : false,
        "default_lang" : "",
        "login_page" : {},
        "nav" : {},
        "uptime" : {},
        "portal_section" : {},
        "designer" : {},
        "dont_show_admin_sockets" : false,
        "dont_allow_license_management" : false,
        "dont_allow_license_management_view" : false,
        "cloud" : false
    },
    "org_options_meta" : {}
}

As such, we need to map the Tyk Organization object to a k8s custom resource.

This object will be the foundation to scaffolding users & apis. e.g. we will first create an organization, then use this object to obtain the org_id, to create the operator user in that org.

This way, the operator will be able to control resources across multiple organizations in a multi-tenant setup.

The organization object will be used to Bootstrap Tyk Pro in a CI/CD environment as part of various CRD integration test suites.

bug: security policies are not idempotent

Apply an API
Apply a security policy
Apply the security policy again and you will see that the CR is modified.

image

kubectl apply -f ./config/samples/httpbin_protected.yaml
kubectl apply -f ./config/samples/httpbin_protected_policy.yaml
kubectl apply -f ./config/samples/httpbin_protected_policy.yaml

The second time we apply the policy CR, kubectl should tell us that the policy was unchanged, rather than configured.

This implies that the reconciler is modifying the security policy custom resource.

This may be resolvable by introducing defaulting webhook for security policies: #134

api: enable detailed recording

As the name suggests, this is to add the ability to toggle the Detailed Logging capability for an API.

Field in API definition:
"enable_detailed_recording": false/true

fr: Webhooks

Need the ability to also manage webhooks via CRD.

-I want to create a webhook with a unique name inside a namespace
-CRUD events should target the name/namespace combo
-API definitions that include. webhooks will use the webhook's name/namespace combo.

This creates a new dependency. You should not be able to create an API with a webhook event that targets a webhook that does not exist yet. If that happens, requeue the ApiDefinition renconcile until the webhook events

apidef: introduce udg support

Using ApiDefinition custom resources, it should be possible to stitch APIs together & publish them as a GraphQL API.
It should be extensible enough to support new data-sources as they become available in the underlying libraries.
This is particularly challenging, because each data-source has different configurations. In this regard, we may need to strongly type each data-source type, then recreate the interface or json.RawMessage when calling the Tyk APIs.


Data Sources to support OOTB:

  • External GraphQLData Source
  • External HTTP JSON Source
  • Internal (Tyk-Managed) GraphQLData Source
  • Internal (Tyk-Managed) HTTP JSON Source
  • Extensible to support new data sources as they are introduced to underlying library, whilst maintaining benefit of strong typing & validation

  • introduce new types if needed
  • generate CRDs
  • documentation in context of ./config/samples
  • README example
  • tests

Non-Goals:

  • Security Policies for GraphQL protection will be addressed as a separate feature.

Example:

apiVersion: tyk.tyk.io/v1alpha1
kind: ApiDefinition
metadata:
  name: composed
spec:
  name: composed
  use_keyless: true
  protocol: http
  active: true
  proxy:
    target_url: https://countries.trevorblades.com
    listen_path: /composed
    strip_listen_path: true
  graphql:
    enabled: true
    execution_mode: executionEngine
    schema: |                                                  <----------------- VALIDATE THIS SCHEMA INSIDE VALIDATING WEBHOOK
      type Country {
        name: String
        code: String
        enriched: RESTCountry
      }
      type Query {
        countries: [Country]
      }
      type RESTCountry {
        subregion: String
        population: Int
      }
    type_field_configurations:
      - type_name: Query
        field_name: countries
        data_source:
          kind: GraphQLDataSource
          graphql_data_source:
            method: POST
            url: https://countries.trevorblades.com
        mapping:
          disabled: true
          path: ""
      - type_name: Country
        field_name: enriched
        data_source:
          kind: HTTPJSONDataSource
          http_json_data_source:
            default_type_name: "RESTCountry"
            method: GET
            status_code_type_name_mappings:
              - status_code: 200
            url: https://restcountries.eu/rest/v2/alpha/{{ .object.code }}
        mapping:
          disabled: true
          path: ""
    playground:                           <----------------- ENABLE GRAPHQL PLAYGROUND (SAME / SHARED WITH GRAPHQL PROXY)
      enabled: true
      path: /playground

bug: permissions issue with events

Create an api definition then delete it. Check the tyk-operator logs. Looks like we need a ClusterRole, not just a Role.

https://github.com/TykTechnologies/tyk-operator/blob/master/helm/templates/all.yaml#L1-L33

2020-11-16T20:04:30.053Z        DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"ApiDefinition","namespace":"tykpro-control-plane","name":"httpbin","uid":"2d966b3a-8c90-4444-836b-91401db05954","apiVersion":"tyk.tyk.io/v1alpha1","resourceVersion":"11238"}, "reason": "ApiDefinition", "message": "Reconciling"}
E1116 20:04:30.060985       1 event.go:260] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"httpbin.1648154326014460", GenerateName:"", Namespace:"tykpro-control-plane", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ApiDefinition", Namespace:"tykpro-control-plane", Name:"httpbin", UID:"2d966b3a-8c90-4444-836b-91401db05954", APIVersion:"tyk.tyk.io/v1alpha1", ResourceVersion:"11238", FieldPath:""}, Reason:"ApiDefinition", Message:"Reconciling", Source:v1.EventSource{Component:"apidefinition-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfe4d44952031260, ext:91820940701, loc:(*time.Location)(0x2442120)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfe4d4b3832fc1fc, ext:517010894401, loc:(*time.Location)(0x2442120)}}, Count:4, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "httpbin.1648154326014460" is forbidden: User "system:serviceaccount:tyk-operator-system:tyk-operator" cannot patch resource "events" in API group "" in the namespace "tykpro-control-plane"' (will not retry!)

ingress without template should create simple keyless api definition

It should not be mandatory to apply a template API Definition.
The most basic ingress object should be enough to create a keyless API Definition.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin-ingress
  annotations:
    kubernetes.io/ingress.class: tyk
spec:
  rules:
    - host: httpbin.ahmet
      http:
        paths:
          - path:
            pathType: Prefix
            backend:
              service:
                name: httpbin
                port:
                  number: 8000

At the moment, the template annotation in the ingress route is mandatory:

	// ensure there is a template api definition
	templateAnnotationValue, ok := desired.Annotations[ingressTemplateAnnotationKey]
	if !ok {
		return ctrl.Result{}, fmt.Errorf("expecting template annotation %s", ingressTemplateAnnotationKey)
	}

This causes issues with cert-manager creating ingress resources as we currently require:

# annoying annotations which means if the API Definitions are protected (not keyless), then we cannot enable TLS
acme.cert-manager.io/http01-edit-in-place: "true"
# funky hacks to handle the acme-http challenge

			if !strings.Contains(p.Path, ".well-known/acme-challenge") && !strings.Contains(p.Backend.ServiceName, "cm-acme-http-solver") {
				for _, tls := range desired.Spec.TLS {
					for _, host := range tls.Hosts {
						if hostName == host {
							api.Spec.Protocol = "https"
							api.Spec.CertificateSecretNames = []string{
								tls.SecretName,
							}
							api.Spec.ListenPort = 443
						}
					}
				}
			} else {
				// for the acme challenge
				api.Spec.Proxy.StripListenPath = false
				api.Spec.Proxy.PreserveHostHeader = true
			}

Referencing objects that already exist

What happens if I want to migrate to the Operator pattern, but not recreate all my Tyk objects in CRD as to not disrupt my current users. What if I want to re-use my organization, api_id etc ?

For example, I have a Tyk install running. I have users using my APIs in production. I don't want to recreate my existing APIs in CRs so as to not disrupt my existing keys. But I want to create a new policy via the CRD. In this case, we would need a way to specify in the CR to tell the Operator which API or Policy or Webhook or Organization we are referencing.

Perhaps an annotation or a bool flag to say, "this API already exists, here's the BSON ID". here's an example of a securitypolicy CR referencing an existing API:

apiVersion: tyk.tyk.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: httpbin
spec:
  name: Some other Security Policy
  state: active
  active: true
  access_rights_array:
    - provide_mongo_id: true
      bson_id: <api_id>

policies (feat req): API level limits

requesting the ability to add per API level limits into a SecurityPolicy declaratively.

Here's a screenshot from the Tyk Dashboard that illustrates this.
image

This functionality allows you to do per API limits in a policy.
Here's the field that needs to be enabled to allow this to happen.

ci: deploy Tyk Pro for CI integration testing

Bootstrap a Tyk Pro installation for integration testing of CRs.
We should have 2 gateway clusters in different namespaces. That way we can tag our API Definitions ensuring that we can deploy different APIs to each cluster.

Namespace:

  • tykpro-control-plane

Dependencies:

  • Redis
  • MongoDB
  • Dashboard
  • Gateway * 2

Bootstrap:

  • Dashboard requires bootstrapping organization
  • Initial operator user - to obtain dashboard API Key (permissions for dash api)
  • Deploy CRDs to the cluster
  • Scaffold Test suite to run through CRUD of API Definitions + CURL the gateway to test responses

feat: support gRPC plugins

Feature: Support gRPC plugins in ApiDefinition custom resource
  In order to customize the middleware chain
  As a developer
  I need to be able to configure an api to use gRPC plugins

  @undone
  Scenario: Pre middleware hook
    Given there is a "TODO: KEYLESS WITH PRE HOOK" resource
    When i request /httpbin/headers endpoint
    Then there should be a 404 http response code
      And the response should match json:
      """
      {
        "todo": "something which shows gRPC plugin short-circuited at the pre hook"
      }
      """

  @undone
  Scenario: Auth middleware hook unauthenticated
    Given there is a "TODO: gRPC AUTH" resource
    When i request /httpbin/get endpoint
    Then there should be a 401 http response code

  @undone
  Scenario: Auth middleware hook authentic
    Given there is a "TODO: gRPC AUTH" resource
    When i request /httpbin/get endpoint with authorization header "SOMEAUTHTOKEN"
    Then there should be a 200 http response code

  @undone
  Scenario: Auth middleware with ID Extractor
    Given there is a "TODO: gRPC AUTH" resource
    When i request /httpbin/get endpoint with authorization header "SOMESTATICAUTHTOKEN"
      And i request /httpbin/get endpoint with authorization header "SOMESTATICAUTHTOKEN"
    Then there should be a 200 http response code
      And the second response should be quicker than the first response

  @undone
  Scenario: Post auth middleware hook
    Given there is a "AuthToken" resource
    When i request "/httpbin/get" endpoint with authorization header "SOMESTATICAUTHTOKEN"
    Then there should be a 200 http response code
      And the response should match json:
      """
      {
        "todo": "something which shows gRPC plugin short-circuited at the post auth hook"
      }
      """

  @undone
  Scenario: Post middleware or pre-proxy hook
    Given there is a "keyless" resource
    When i request "/httpbin/get" endpoint with authorization header "SOMESTATICAUTHTOKEN"
    Then there should be a 200 http response code
    And the response should match json:
      """
      {
        "todo": "something which shows gRPC plugin short-circuited at the post hook"
      }
      """

In order to develop this feature, we need

  • a gRPC plugin which can handle pre, auth, post_auth and post hooks.
  • The plugin will need to be available as docker image, and loaded into the CI. https://hub.docker.com/r/mangomm/tyk-grpc-plugin
  • OSS & Pro Gateway configs (CI) need to be configured to use the gRPC plugin server in CI environment.
  • Sample API Definitions for each of the scenarios above.

track: gateway hello endpoint not always available

When you split the control port and the api listen port, the /hello endpoint becomes problematic.

For example, if you delete ALL apis, then the /hello endpoint disappears - a load balancer cannot target this endpoint because the gateway stops listening on that port.

If you enable host based routing, e.g. your api listen path is on /, then this causes conflicts with the /hello endpoint.

Ramifications for this setup means that a cloud load balancer will switch to unhealthy, because it is unable to target the /hello endpoint. This makes configuring ingress for cloud environments difficult. However it is possible if we hack it, by creating a dummy HTTP API Definition. listening on the same port as the hello endpoint.

I think that the /hello endpoint belongs with the control APIs / control port

bug: deleting all apis in the gateway should return 404 not found.

This looks like a Gateway bug, however after investigation & resolution, we may need to remove the workaround in the tests.

This occurs with Tyk Pro, not Tyk CE.
And the control port needs to be different to the proxy port.

If I create an API - e.g. httpbin, then delete it - I expected to get back a 404 not found when requesting the API. Rather, the gateway seems to stop listening on the port entirely.

Maybe this is desired behaviour - however it seems that Tyk CE acts differently. returning a 404 not found.

$ kubectl port-forward -n tykpro-control-plane svc/gw 8000:8000
E1102 09:43:14.530780   34478 portforward.go:400] an error occurred forwarding 8000 -> 8000: error forwarding port 8000 to pod e441c00617758a1233fff01b7d93fa39339eec1fcaef508df85ebf744c4dc404, uid : exit status 1: 2020/11/02 09:43:14 socat[8642] E connect(5, AF=2 127.0.0.1:8000, 16): Connection refused

https://github.com/TykTechnologies/tyk-operator/blob/master/bdd/godogs_test.go#L128-L136

$ curl localhost:8000/httpbin/get
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip", 
    "Host": "httpbin.default.svc:8000", 
    "User-Agent": "curl/7.71.1"
  }, 
  "origin": "127.0.0.1", 
  "url": "http://httpbin.default.svc:8000/get"
}
$ k delete tykapis httpbin
apidefinition.tyk.tyk.io "httpbin" deleted
$ curl localhost:8000/httpbin/get
curl: (52) Empty reply from server

track: deleting API leaves artifacts inside the organization document

https://tyktech.atlassian.net/browse/TT-855

Creating an API using the tyk-operator creates 2 artifacts inside the organization object.

The Original API ID, and the Custom API ID.

Screenshot 2020-10-07 at 14 34 57

When deleting the API by the custom ID, the original ID remains as artifact.

Screenshot 2020-10-07 at 14 37 48


I think that the problem is not necessarily with Deleting an API, but the act of Updating the API (to provide a custom ID) fails to update the organization object with the new reference and just creates a new entry.

docs: active flag in ApiDefinition resource is Pro feature

Tyk Pro allows us to create "draft" APIs by marking the ApiDefinition object as active: false.

When the Gateway pulls the API Definitions from the dashboard API, it doesn't get the active: false APIs, only the active: true ones.

As such, the OSS Gateway does not have any support for creating an inactive API.

This should be documented in: https://github.com/TykTechnologies/tyk-operator/blob/master/docs/api_definitions.md#features


It would be great to have a "pro" icon, emoji or similar where we can highlight features which are only available in licensed installations. Or a warning emoji in the feature comments specifying that it is a pro-only feature.

Maybe we can add the :godmode: emoji :godmode: to the key to represent a Pro / licensed only feature?

Store dependencies in ConfigMap

Need to store the dependencies between the different objects in ConfigMaps, so that object delete requests can be checked in O(1) fashion. currently, we are iteratively doing it by combing through all Policies & Apis and doing comparisons. Not ideal and also hard to visually see the dependencies this way.

  • (1) API can't be deleted if a security policy contains it, will continue to reconcile
  • (2) API can't be created if references policy that does not exist
    (chicken and egg, how to handle scenario where API has policy (such as OIDC case), and both are created at same time? How does reconciler work?)
  • (3) Webhook cant be deleted if an API contains it, will continue to reconcile
  • (4) SecurityPolicy can't be created if references API that does not exist
  • (5) None of the above can be created without Organization

2 ConfigMaps (per namespace), example:

- tyk_object_dependencies
---- webhooks/my-sample-webhook = ["my-httpbin-api"]
---- api/my-httpbin-api = ["my-security-policy-httpbin"]

example flow of API (1)

  1. create API
  2. create security policy which reference that api
  3. attempt to delete api
    A) operator checks api in the map, sees api/my-httpbin-api = ["my-security-policy-httpbin"], turns out this API is being used in a policy, cannot be deleted. need to delete policy first

example flow of API (2)

  1. create API with JWT where default policy is one that does not exist
    should fail creation and requeue

example flow of (3)

  1. create webhook
  2. create APi which reference that webhook
  3. attempt to delete webhook
    A) operator checks webhook in the map, sees webhooks/my-sample-webhook = ["my-httpbin-api"], turns out this webhook is being used, cannot be deleted. need to delete webhook first

apidef: introduce GraphQL proxy support

  • introduce new types
  • generate CRDs
  • documentation in context of ./config/samples
  • README example

Non-Goals:

  • UDG is out of scope & will be addressed as a separate feature.
  • Security Policies for GraphQL protection will be addressed as a separate feature.

bug: unable to deploy operator inside cluster when webhooks enabled

It is not currently possible to deploy the operator due to what appears to be some Kubebuilder missing namespace compatibility issue.

make deploy IMG=tykio/tyk-operator:latest                                                                                                        
/home/asoorm/go/bin/controller-gen "crd:trivialVersions=true,crdVersions=v1" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && /home/asoorm/go/bin/kustomize edit set image controller=tykio/tyk-operator:latest
/home/asoorm/go/bin/kustomize build config/default | kubectl apply -f -
Error: no matches for OriginalId admissionregistration.k8s.io_v1beta1_MutatingWebhookConfiguration|~X|mutating-webhook-configuration; no matches for CurrentId admissionregistration.k8s.io_v1beta1_MutatingWebhookConfiguration|~X|mutating-webhook-configuration; failed to find unique target for patch admissionregistration.k8s.io_v1beta1_MutatingWebhookConfiguration|mutating-webhook-configuration
error: no objects passed to apply
make: *** [Makefile:57: deploy] Error 1

Workaround for time being is to ship without Defaulting / Validating capabilities - which is a BIG shame unless we can resolve this.

See:

@excieve could you take a look for me please?

wh: configure webhooks to catch events that occur at API level

Create a webhook custom resource which can CRUD webhooks in a similar manner to API Definitions & Security Policies.

{
    "_id" : ObjectId("5f7c71617250af90cf5861c5"),
    "org_id" : "5d67b96d767e02015ea84a6f",
    "name" : "foo",
    "method" : "GET",
    "target_path" : "https://webhook.site/12cc63fd-f890-4fda-b05d-c444881f2795",
    "template_path" : "",
    "header_map" : {},
    "event_timeout" : NumberLong(0)
}

Webhooks should be targetable by namespace / name rather than having to select the webhook to fire using the mongo id 5f7c71617250af90cf5861c5.

  "hook_references": [
    {
      "event_name": "AuthFailure",
      "event_timeout": 60,
      "hook": {
        "method": "GET",
        "api_model": {},
        "header_map": {},
        "template_path": "",
        "name": "foo",
        "target_path": "https://webhook.site/12cc63fd-f890-4fda-b05d-c444881f2795",
        "org_id": "5d67b96d767e02015ea84a6f",
        "id": "5f7c71617250af90cf5861c5",
        "event_timeout": 0
      }
    }
  ]

Why is the webhook object is copied into the API Definition object.
If a webhook obj changes, does this get transposed to API Definitions also, or does it leave the API Definition in an inconsistent state?
Is it possible to configure webhooks in the API Definition without having created a webhook?
What fields in the webhook are mandatory?

feat: cut a release

https://github.com/TykTechnologies/tyk-operator/actions?query=workflow%3A%22Cut+Release%22

https://github.com/TykTechnologies/tyk-operator/blob/master/.github/workflows/cut_release.yml

We have a new GitHub action - called cut_release.

It is triggered manually from GitHub Actions.

Screenshot 2020-12-02 at 13 41 04

We specify the version of the release we want to cut - and apply that to the master branch.

When we manually trigger the release, the action should:

This will trigger a goreleaser github action / build pipeline to push to Dockerhub - once master branch is pushed.


The current flow is a little manual & as such, if i forget to bump the Chart.yaml or something - then my release is broken.

feat: upgrade crds and webhooks to v1

After deploying the operator using make deploy IMG=tykio/tyk-operator:latest, then applying an API definition:

kubectl apply -f ./config/samples/httpbin.yaml

Error from server (InternalError): error when creating "./config/samples/httpbin.yaml": Internal error occurred: failed calling webhook "mapidefinition.kb.io": expected response.uid="c27dafd5-51e9-4b21-b773-9da861d0a14e", got ""

Operator logs:

2020-10-25T17:48:51.371Z        ERROR   controller-runtime.webhook.webhooks     unable to decode the request    {"webhook": "/mutate-tyk-tyk-io-v1alpha1-apidefinition", "error": "no kind \"AdmissionReview\" is registered for version \"admission.k8s.io/v1\" in scheme \"pkg/runtime/scheme.go:101\""}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:79
sigs.k8s.io/controller-runtime/pkg/webhook.instrumentedHook.func1
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/server.go:129
net/http.HandlerFunc.ServeHTTP
        /usr/local/go/src/net/http/server.go:2036
net/http.(*ServeMux).ServeHTTP
        /usr/local/go/src/net/http/server.go:2416
net/http.serverHandler.ServeHTTP
        /usr/local/go/src/net/http/server.go:2831
net/http.(*conn).serve
        /usr/local/go/src/net/http/server.go:1919
2020-10-25T17:48:51.372Z        DEBUG   controller-runtime.webhook.webhooks     wrote response  {"webhook": "/mutate-tyk-tyk-io-v1alpha1-apidefinition", "UID": "", "allowed": false, "result": {}, "resultError": "got runtime.Object without object metadata: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:no kind \"AdmissionReview\" is registered for version \"admission.k8s.io/v1\" in scheme \"pkg/runtime/scheme.go:101\",Reason:,Details:nil,Code:400,}"}

feat: url rewrite to internal APIs

Dynamic Auth

In this example, we have a bunch of legacy customers who authenticate with our service using Basic Authentication.
We want to be able to support API Keys also, where both types of clients hit the same ingress.
As such, Tyk needs to decide whether to perform Basic Authentication or Auth Token auth check.

In order to achieve this, we need to configure 4 API Definitions inside Tyk.

  1. EntryPoint API
  2. BasicAuthInternal API
  3. AuthTokenInternal API
  4. ProxyInternal API

When the request hits the ingress route, we configure a URL rewrite to pass the request to either the internal
BasicAuth api or the AuthToken API.
the internal APIs will then authenticate request, and assuming the happy path, proxy to the ProxyInternal API.
The ProxyInternal API is responsible for proxying to the underlying service.

It is worth noting that there are no actual http redirect happening here, meaning that there is no performance penalty
in performing any of these "Internal Redirects".

Rather than having to specify the APIID, to proxy to, we should be able to target the namespace/name of the APIDefinition custom Resource.

If the BasicAuthInternal or ProxyInternal custom resources are attempted to be deleted, then the finalizer should fail & retry till the API(s) which are referencing it are either also deleted or de-reference.

---
apiVersion: tyk.tyk.io/v1alpha1
kind: ApiDefinition
metadata:
  name: entrypoint-api
spec:
  name: Entrypoint API
  protocol: http
  active: true
  proxy:
    listen_path: /entrypoint
    strip_listen_path: true
    # TODO: We should be able to create ROUTES without a target URL.
    target_url: http://dummy.com
  use_keyless: true
  # if Authorization: Basic - redirect to tyk://APIID where APIID is the ID associated with namespace/name e.g. default/basic-auth-internal
  # if Authorization: Bearer - redirect to tyk://APIID where APIID is the ID associated with namespace/name e.g. default/auth-token-internal
---
apiVersion: tyk.tyk.io/v1alpha1
kind: ApiDefinition
metadata:
  name: basic-auth-internal
spec:
  name: BasicAuth Internal API
  protocol: http
  active: true
  internal: true
  use_basic_auth: true
  auth_configs:
    basicAuth:
      auth_header_name: Authorization
  version_data:
    default_version: Default
    not_versioned: true
    versions:
      Default:
        name: Default
        # redirect all traffic to proxy API
---
apiVersion: tyk.tyk.io/v1alpha1
kind: ApiDefinition
metadata:
  name: auth-token-internal
spec:
  name: AuthToken Internal API
  protocol: http
  active: true
  internal: true
  use_basic_auth: true
  auth_configs:
    authToken:
      auth_header_name: Authorization
  version_data:
    default_version: Default
    not_versioned: true
    versions:
      Default:
        name: Default
        # redirect all traffic to proxy API
---
apiVersion: tyk.tyk.io/v1alpha1
kind: ApiDefinition
metadata:
  name: proxy-api
spec:
  name: Proxy API
  protocol: http
  active: true
  internal: true
  proxy:
    target_url: http://httpbin.org
  use_keyless: true
---

@buger FYI

mw: JSON Schema Validation support

Tyk Dashboard API expects the JSON Schema Validation schema to be presented as a JSON object.
Tyk Gateway API expects the JSON Schema Validation shema JSON to be presented as a b64 encoded string.

In this regard, the Tyk Operator with OSS Gateway will work, however because we cannot define arbitrary JSON map[string]interface{} as a Kubernetes CRD, it is currently not possible to support JSON Schema in a pro Tyk installation.

In this regard, one proposal would be to have an apidefinition_types.go and a dashboard_apidefinition_types, with a mediation layer to convert between them.

An alternative would be to have a custom marshaller & unmarshaller for the map[string]interface{}.

research: support ingress resources

tyk-operator should be usable as an ingress controller.

Proposal:

# example ingress resource
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test.01.ingress
spec:
  ingressClassName: "test.01.ingress.class"
  rules:
    - http:
        paths:
          # creates 2 APIDefinitions to configure the Gateway, using the template acquired from IngressClass: test.01.ingress.class
          - path: /httpbin
            pathType: Prefix
            backend:
              service:
                name: httpbin
                port:
                  number: 8000
          - path: /petstore
            pathType: Prefix
            backend:
              service:
                name: petstore
                port:
                  number: 8000
---

# example 2nd ingress resource
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test.02.ingress
spec:
  # multiple ingress resources can share the same ingress class
  ingressClassName: "test.01.ingress.class"
  rules:
    - http:
        paths:
          - path: /trevorblades
            pathType: Prefix
            backend:
              service:
                name: trevorblades
                port:
                  number: 8000
---
# example ingress class resource
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: test.01.ingressclass
spec:
  # this ingress class is managed by tyk-operator
  controller: tyk-operator
  parameters:
    apiGroup: tyk
    kind: ApiDefinition
    name: http-ingress-template
---
# example template apidef for http
apiVersion: tyk.tyk.io/v1alpha1
kind: ApiDefinition
metadata:
  name: http-ingress-template
  # do we use labels or annotations?
  # this means apidefinition reconciler will just return `ctrl.Result{}, nil`
  labels:
    name: template
  annotations:
    ingressClass: "true"
spec:
  name: template
  protocol: http
  listen_port: 80
  proxy:
    # placeholder stuff here
    listen_path: /dummy
    target_url: http://dummy.com
---
# example template apidef for https
apiVersion: tyk.tyk.io/v1alpha1
kind: ApiDefinition
metadata:
  name: https-ingress-template
  # do we use labels or annotations?
  # this means apidefinition reconciler will just return `ctrl.Result{}, nil`
  labels:
    name: template
  annotations:
    ingressClass: "true"
spec:
  name: template
  protocol: http
  listen_port: 443
  proxy:
    # placeholder stuff here
    listen_path: /dummy
    target_url: http://dummy.com

synchronise certs stored in K8s secrets into the Tyk API Manager

When cert-manager creates a certificate, it is stored inside a secret with the following properties:

Type: "kubernetes.io/tls"
Data:
  - tls.crt
  - tls.key

Implement a controller which watches secrets, and looks for the type kubernetes.io/tls as part of it's predicate filter.
In the reconcile function - we should determine whether the secret is being created / updated / deleted, and mirror that inside Tyk's Certificate Store.

Tyk stores certificates with an id as follows: ${ORG_ID}${CERT_FINGERPRINT}. It is this ID which needs to be eventually injected into the API Definition object.

Injecting certificates into API Definitions is part of the role of the apidefinition_reconciler. This will need an update.

Feature: Managing https APIs
  In order to terminate TLS, Tyk operator
  must be able to load, rotate and delete 
  TLS certificates stored as secrets
  into Tyk's certificate manager.
  
  @undone
  Scenario: Create a certificate
    Given there is a "self-signed-issuer" resource
      And there is a "certificate" resource
      And there is a "https" resource
    Then the admin "/certs/ID" should be created
      And the "https" resource should reference the new certificate

  @undone
  Scenario: Rotate/Renew a certificate
    Given there is a "self-signed-issuer" resource
      And there is a "certificate" resource
      And there is a "https" resource
    When i kubectl cert-manager renew
    Then the admin "/certs/ID" should be created
      And the "https" resource should reference the new certificate

  @undone
  Scenario: Delete a certificate
    Given there is a "self-signed-issuer" resource
      And there is a "certificate" resource
      And there is a "https" resource
    When i delete the "certificate" resource
      And i delete the "secret" resource
    Then the admin "/certs/ID" should be deleted
      And the "https" resource should not reference any certificates

api resource is created even when there is error with universal client

I came across this when sending APIDefinition object to the dashboard without the Auth field. The universal client returns a validation error from the dashboard, however the reconciler logic doesn't return the error, it only logs it and returns nil.

The line of interest is here

log.Error(err, "createOrUpdate failure")
return ctrl.Result{Requeue: true}, nil

Expected bahavior

An error should be returned and the resource should not be created

make helm should be interoperable with Mac & Linux

At present, make helm is incompatible with BSD sed. This makes it impossible to work with on Linux machines.

We should detect whether GNU sed or BSD sed is installed on host machine & run appropriate version.

We could also find an alternative to sed which is interoperable - meaning that it should not matter whether helm templates are generated from Linux / Mac. e.g. use Kustomize to fully generate the helm templates meaning we don't need to rely on sed at all.

helm: kustomize
	$(KUSTOMIZE) build config/crd > ./helm/crds/crds.yaml
	$(KUSTOMIZE) build config/helm > ./helm/templates/all.yaml
	sed -i '' 's#replicas: 1#replicas: {{ \.Values.replicaCount }}#' helm/templates/all.yaml
	sed -i '' 's#tyk-operator-conf#{{ \.Values\.confSecretName }}#' helm/templates/all.yaml
	sed -i '' 's#tykio/tyk-operator:latest#{{ \.Values\.image\.repository }}:{{ \.Values\.image\.tag }}#' helm/templates/all.yaml
	sed -i '' 's#imagePullPolicy: IfNotPresent#imagePullPolicy: {{ .Values.image.pullPolicy }}#' helm/templates/all.yaml
	sed -i '' 's#name: default#name: {{ include "tyk-operator-helm\.serviceAccountName" \. }}#' helm/templates/all.yaml
	sed -i '' 's#serviceAccountName: default#serviceAccountName: {{ include "tyk-operator-helm\.serviceAccountName" \. }}#' helm/templates/all.yaml

Store Mongo IDs for Tyk Pro objects in CR status field

We need to store the IDs for generated objects in Tyk Pro on the CR, so that we can use this to do further CRUD operations down the line. Right now, we are fetching entire lists and comparing each item. This does not scale well at all

research: how to package & deploy the Tyk Operator

There are many ways in which an operator can be deployed into a cluster:

  • YAMLs
  • Operator Lifecycle Manager OpenShift specific
  • Kustomize
  • Helm

Maybe at some stage, we will support them all, however for MVP, we need to pick a method & prioritise order in which we will introduce each deployment method.

Would be great to get some voting in place to understand priorities.


Helm seems the obvious solution, however it is not without it's pitfalls: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations

There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.