Coder Social home page Coder Social logo

workflow's Introduction

KubeVela Workflow

Go Report Card codecov LICENSE Total alerts

What is KubeVela Workflow

KubeVela Workflow is an open-source cloud-native workflow project that can use to orchestrate CI/CD process, terraform resources, multi-kubernetes-clusters management and even your own functional calls.

You can install KubeVela Workflow and use it, or import the code as an sdk of an IaC-based workflow engine in your own repository.

The main differences between KubeVela Workflow and other cloud-native workflows are:

All the steps in the workflow is based on IaC(Cue): every step has a type for abstract and reuse, the step-type is programmed in CUE language and easy to customize.

That is to say, you can use atomic capabilities like a function call in every step, instead of just creating a pod.

Why use KubeVela Workflow

workflow arch

🌬️ Lightweight Workflow Engine: KubeVela Workflow won't create a pod or job for process control. Instead, everything can be done in steps and there will be no redundant resource consumption.

Flexible, Extensible and Programmable: Every step has a type, and all the types are based on the CUE language, which means if you want to customize a new step type, you just need to write CUE codes and no need to compile or build anything, KubeVela Workflow will evaluate these codes.

💪 Rich built-in capabilities: You can control the process with conditional judgement, inputs/outputs, timeout, etc. You can also use the built-in step types to do some common tasks, such as deploy resources, suspend, notification, step-group and more!

🔐 Safe execution with schema checksum checking: Every step will be checked with the schema, which means you can't run a step with a wrong parameter. This will ensure the safety of the workflow execution.

Try KubeVela Workflow

Run your first WorkflowRun to distribute secrets, build and push your image, and apply the resources in the cluster! Image build can take some time, you can use vela workflow logs build-push-image --step build-push to check the logs of building.

apiVersion: core.oam.dev/v1alpha1
kind: WorkflowRun
metadata:
  name: build-push-image
  namespace: default
spec:
  workflowSpec:
   steps:
   # or use kubectl create secret generic git-token --from-literal='GIT_TOKEN=<your-token>'
    - name: create-git-secret
      type: export2secret
      properties:
        secretName: git-secret
        data:
          token: <git token>
    # or use kubectl create secret docker-registry docker-regcred \
    # --docker-server=https://index.docker.io/v1/ \
    # --docker-username=<your-username> \
    # --docker-password=<your-password> 
    - name: create-image-secret
      type: export2secret
      properties:
        secretName: image-secret
        kind: docker-registry
        dockerRegistry:
          username: <docker username>
          password: <docker password>
    - name: build-push
      type: build-push-image
      properties:
        # use your kaniko executor image like below, if not set, it will use default image oamdev/kaniko-executor:v1.9.1
        # kanikoExecutor: gcr.io/kaniko-project/executor:latest
        # you can use context with git and branch or directly specify the context, please refer to https://github.com/GoogleContainerTools/kaniko#kaniko-build-contexts
        context:
          git: github.com/FogDong/simple-web-demo
          branch: main
        image: fogdong/simple-web-demo:v1
        # specify your dockerfile, if not set, it will use default dockerfile ./Dockerfile
        # dockerfile: ./Dockerfile
        credentials:
          image:
            name: image-secret
        # buildArgs:
        #   - key="value"
        # platform: linux/arm
    - name: apply-deploy
      type: apply-deployment
      properties:
        image: fogdong/simple-web-demo:v1

Quick Start

After installation, you can either run a WorkflowRun directly or from a Workflow Template. Every step in the workflow should have a type and some parameters, in which defines how this step works. You can use the built-in step type definitions or write your own custom step types.

Please checkout the WorkflowRun Specification and WorkflowRun Status for more details.

Run a WorkflowRun directly

For more, please refer to the following examples:

Run a WorkflowRun from a Workflow Template

Please refer to the following examples:

Installation

Install Workflow

Helm

helm repo add kubevela https://kubevela.github.io/charts
helm repo update
helm install --create-namespace -n vela-system vela-workflow kubevela/vela-workflow

KubeVela Addon

If you have installed KubeVela, you can install Workflow with the KubeVela Addon:

vela addon enable vela-workflow

Install Vela CLI(Optional)

Please checkout: Install Vela CLI

Features

Step Types

Built-in Step Types

Please checkout the built-in step definitions with scope that valid in WorkflowRun.

Write Your Custom Step Types

If you're not familiar with CUE, please checkout the CUE documentation first.

You can customize your steps with CUE and some built-in operations. Please checkout the tutorial for more details.

Note that you cannot use the application operations since there're no application data like components/traits/policy in the WorkflowRun.

How can KubeVela Workflow be used

During the evolution of the OAM and KubeVela project, workflow, as an important part to control the delivery process, has gradually matured. Therefore, we separated the workflow code from the KubeVela repository to make it standalone. As a general workflow engine, it can be used directly or as an SDK by other projects.

As a standalone workflow engine

Unlike the workflow in the KubeVela Application, this workflow will only be executed once, and will not keep reconciliation, no garbage collection when the workflow object deleted or updated. You can use it for one-time operations like:

  • Glue and orchestrate operations, such as control the deploy process of multiple resources(e.g. your Applications), scale up/down, read-notify processes, or the sequence between http requests.
  • Orchestrate delivery process without day-2 management, just deploy. The most common use case is to initialize your infrastructure for some environment.

As an SDK

You can use KubeVela Workflow as an SDK to integrate it into your project. For example, the KubeVela Project use it to control the process of application delivery.

You just need to initialize a workflow instance and generate all the task runners with the instance, then execute the task runners. Please check out the example in Workflow or KubeVela.

Contributing

Check out CONTRIBUTING to see how to develop with KubeVela Workflow.

workflow's People

Contributors

arvintian avatar barnettzqg avatar chaitanyareddy0702 avatar chivalryq avatar dependabot[bot] avatar fogdong avatar jongwooo avatar lijiamu233 avatar somefive avatar wangyikewxgm avatar wonderflow avatar yangsoon avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

workflow's Issues

[Feature]Add version management to Cue Actions

Is your feature request related to a problem? Please describe.

The current Cue Action lacks version management. Whenever the parameters of the Action are changed, it will affect the previous users.

Describe the solution you'd like

We can store all different versions of Cue Action in the directory, and the folders name follow the rules below:

  1. For the stable version, the folder name and pkgName is “vela/op”, which is the same as before in order to the previous users can still use their Definition without editing.
  2. For other versions, the name is “vela/op/v2”,”vela/op/v3” etc. Users need to import them explicitly and change the “op.action” to “v2.action” and so on to use another version of Cue Action.

image

Firstly, embed all files and directories to the file system and load content of each Action folder into strings.Then generate multiple instances corresponding to different strings with different PkgName and ImportPath, then call ParseFile and AddSyntax to build. Finally return a set of "BuiltinImports"

Describe alternatives you've considered

Additional context

[Feature] Add suspend for step but not the whole workflow

Is your feature request related to a problem? Please describe.

Currently the suspend will affect the whole workflow, we need this suspend to more specific to let one step suspend but not affect the whole workflow.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Seek usage help

hello, I recently learn "kubevela/workflow", your work is so great.
But there are some nuts when I use the builtin function "#ApplyInParallel"
Function: Input a volume name list, and I want to add snapshot on these volumes by adding a field in "spec".【this operator has been defined】

This is the cue template code:

import (
	"vela/op"
	"encoding/json"
)

"add-warpdrive-snapshot": {
	alias: ""
	annotations: {}
	attributes: {}
	description: ""
	labels: {}
	type: "workflow-step"
}

template: {
	output: {
		 data:  op.#Steps & {
		   	  wvs: [ for idx,wv in parameter.warpdrive_list {
		   	  	 op.#Read & {
		   	  	 	value : {
						kind: "WarpdriveVolume"
    						apiVersion: "tos.trans.io/v1"
     					 	metadata: name: wv
     					 }
							}
		   	 	 }]  @step(1)

		   	  log10086: op.#Log & { data: wvs[0] }  @step(2)

          patches: [  for v in wvs {
          	    op.#PatchK8sObject & {
                                value: v.value
                                patch: {
                                	// +patchStrategy=retainKeys
                                	spec: snapshots: ["flowsnapshot"]
                                }
												 }
                } ] @step(3)

        objects: [ for patchResult in patches {patchResult.result}]  @step(4)


        log10087: op.#Log & {
        	data: json.Marshal(patches[0])
        }  @step(5)

        apply: op.#ApplyInParallel & {
		value: objects
						} @step(6)

        }

	}
	parameter: {
		warpdrive_list: [string]
	}
}

The result was that there was no error reported and the "WorkflowRun" run successfully. But there were also no logs reported which I have configured in the code. And the filed was not be added as expectation. So the operation "patch and apply" were failed.

Did I misunderstand the usage of #Steps or other builtin function ? pls help me

[Feature]Support user-defined custom providers

Is your feature request related to a problem? Please describe.

Currently, we support some built-in providers for users to use like Apply, HTTP, etc.

We should also support user-defined custom providers for extensibility.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

[Feature] Support resume operation in pipeline/workflowrun

Is your feature request related to a problem? Please describe.
Describe the solution you'd like
support resume operation to resume the suspend app in the piepline
A clear and concise description of what you want to happen.
A clear and concise description of any alternative solutions or features you've considered.
Add any other context or screenshots about the feature request here.

[Feature] Support `patch` in `op.#Apply`

Is your feature request related to a problem? Please describe.

Support patch strategy in op.#Apply so that users can patch the specified field when applying the resource.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

执行apply-app报错

我在工作流中添加一个apply-app的节点,内容如下:
image

执行时报错:
image

No kind \"DeleteOptions\" is registered for version \"admissionregistration.k8s.io/v1\" in scheme

Describe the bug

when use the "op.#Delete" for deleting custom resource, there is an error happen:
"no kind \"DeleteOptions\" is registered for version \"admissionregistration.k8s.io/v1\" in scheme \"k8s.io/apiextensions-apiserver/pkg/apiserver/apiserver.go:52\""

To Reproduce

Expected behavior

delete target resources successfully and no error be reported

Screenshots

image

Workflow Version

image

Cluster information

image

Additional context

the same value content for listing resourses using op.#List is correct.

[Feature] why this workflowrun demo failed.

I use this demo
https://github.com/kubevela/workflow/blob/main/examples/workflow-run/apply-applications.yaml

yaml is

apiVersion: core.oam.dev/v1alpha1
kind: WorkflowRun
metadata:
  name: apply-applications
  namespace: default
  annotations:
    workflowrun.oam.dev/debug: "true"
spec:
  workflowSpec:
    steps:
      - name: apply-app2
        type: apply-app
        properties:
          ref:
            name: my-app
            key: application
            type: configMap
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app
  namespace: default
data:
  application: |
    apiVersion: core.oam.dev/v1beta1
    kind: Application
    metadata:
      name: webservice-app2
    spec:
      components:
        - name: express-server2
          type: webservice
          properties:
            image: crccheck/hello-world
            ports:
              - port: 8000

kubevela version : v1.8.1
workflow version: v0.5.0

result:

the workflowrun status is failed

error message :

 run step(provider=kube,do=apply): the body of the request was in an unknown format - accepted media types include: application/json-patch+json, application/merge-patch+json, application/apply-patch+yaml

but webservice-app2 application is running and have a pod express-server2-58f7474c55-txqxk is runing

the next ,The yaml file I use under configMap is in another namespace test

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: webservice-app2
  namespace: test
spec:
  components:
    - name: express-server2
      type: webservice
      properties:
        image: crccheck/hello-world
        ports:
          - port: 8000

the application in namespace test is successed and running ,

why apply-applications workflowrun prompt messaget is unknown format and status is failed

clean-jobs workflow step doesn't clean jobs created with WorkflowRun

Describe the bug
WorkflowRun creates a job that is labeled with workflowrun.oam.dev/name and the underlying pod is labeled with workflow.oam.dev/name this difference means clean-jobs is not deleting the jobs when there aren't any matchingLabels provided.

To Reproduce
Steps to reproduce the behavior:

  1. Apply an WorkflowRun, in this example is an addon but should behave the same way with any
apiVersion: core.oam.dev/v1alpha1
kind: WorkflowRun
metadata:
  name: enable-addons-test
  namespace: vela-system
spec:
  mode: 
  workflowSpec:
    steps:
      - name: Enable chartmuseum
        type: addon-operation
        properties:
          addonName: chartmuseum
          operation: enable
          args:
          - --override-definitions

      - name: Clean
        type: clean-jobs
        if: always
        properties:
          namespace: "vela-system"
  1. The first step creates a job and a pod, notice the difference in the labels between the pod using workflow.* and the job using workflowrun.*
apiVersion: v1
kind: Pod
metadata:
  name: enable-addons-test-qealctrkbx-gx2dr
  generateName: enable-addons-test-qealctrkbx-
  namespace: vela-system
  labels:
    workflow.oam.dev/name: enable-addons-test
    workflow.oam.dev/session: qealctrkbx
   ...
--
apiVersion: batch/v1
kind: Job
metadata:
  name: enable-addons-test-qealctrkbx
  namespace: vela-system
  labels:
    enable-addon.oam.dev: enable-addons-test
    workflowrun.oam.dev/name: enable-addons-test
    workflowrun.oam.dev/namespace: vela-system
    ...
  1. Then the clean-jobs runs and that tries for filter both the job and the pod a single parameter for matchingLabels which results in only the pods being cleaned up
if parameter.labelselector == _|_ {
    matchingLabels: "workflow.oam.dev/name": context.name
}

Expected behavior
It should cleanup both pods and jobs

Additional context
A couple of options when it comes to the fix:

  • should the pod have the workflowrun annotation?
  • or should the clean-jobs delete if is workflow or workflowrun which is a trivial fix

The message attribute of the status for a stepdefinition is skipped when empty

Looks like because we have omitempty for the Message, the status of the step status is skipped if its empty
https://github.com/kubevela/workflow/blob/main/api/v1alpha1/types.go#L215

This causes issue when trying out conditions like mentioned here in the example:
https://github.com/kubevela/workflow/blob/main/examples/multiple-apps.md?plain=1#L22

I believe the original plan was to show an empty string for the message if there was none to show.

Also reported in the public slack here: https://cloud-native.slack.com/archives/C01BLQ3HTJA/p1691106630133239

Expected: The multuple-apps example should work without error

Actual: We see the following error when running the apply-app step:

Steps:
    First Execute Time:  2023-08-07T21:23:50Z
    Id:                  4plf87lwe3
    Last Execute Time:   2023-08-07T21:23:50Z
    Name:                check-app-exist
    Phase:               succeeded
    Type:                read-app
    First Execute Time:  2023-08-07T21:23:50Z
    Id:                  658dfew4xq
    Last Execute Time:   2023-08-07T21:23:50Z
    Message:             pre check error: invalid if value: if: undefined field: message
    Name:                apply-app1
    Phase:               skipped
    Reason:              Skip
    Type:                apply-app
    First Execute Time:  2023-08-07T21:23:50Z
    Id:                  y6e6c69wmb
    Last Execute Time:   2023-08-07T21:23:50Z
    Message:             Suspended by field suspend
    Name:                suspend
    Phase:               suspending
    Reason:              Suspend
    Type:                suspend
  Suspend:               true
  Terminated:            false
Events:                  <none>

Parent step‘s status is “skipped” but its child step was still executed

To Reproduce

yaml

apiVersion: core.oam.dev/v1alpha1
kind: WorkflowRun
metadata:
  name: test-tgt
  namespace: default
  annotations:
    workflowrun.oam.dev/debug: "true"
spec:
  mode:
    steps: StepByStep
    subSteps: StepByStep
  workflowSpec:
    steps:
      - name: mig-group
        type: step-group
        if: input.mock == true
        subSteps:
          - name: read-pod
            type: read-pod
            properties:
              namespace: appworkflow
              podName: hdfsjournalnode-286nr-0

cue:

import (
	"vela/op"
)

"read-pod": {
	alias: ""
	annotations: {}
	attributes: {}
	description: ""
	labels: {}
	type: "workflow-step"
}

template: {
	parameter: {
		podName:   string
		namespace: string
	}
	read: op.#Read & {
		value: {
			kind:       "Pod"
			apiVersion: "v1"
			metadata: {
				name:      parameter.podName
				namespace: parameter.namespace
			}
		}
	}

}

Expected behavior

Screenshots

image

Workflow Version
v0.4.2

Cluster information

image

Why can’t utils.SuspendWorkflow() pause a workflowrun that has a step-group?

Describe the bug

I found that in the previous version of the SDK(vela cli 1.7), SuspendWorkflow() would only set run.Status.Suspend = true.

However, the new version of the SDK(vela 1.9) will set sub-step’s sub Phase to: v1alpha1.WorkflowStepPhaseSuspending. This could lead to the workflowrun Suspending state finally being set to state=Executing in Reconcile().

please tell me why? and how can i use utils.SuspendWorkflow() to stop my workflowrun?

To Reproduce

my demo workflow yaml:

apiVersion: core.oam.dev/v1alpha1
kind: WorkflowRun
metadata:
  name: dependency-ng
  namespace: default
spec:
  mode:
    steps: DAG
  workflowSpec:
    steps:
      - name: step1
        type: apply-deployment
        dependsOn:
          - step2
          - step3
        properties:
          image: 10.113.xx.x:31104/library/nginx:v1.23.2
      - name: step2
        type: apply-deployment
        properties:
          image: 10.113.xx.x:31104/library/nginx:v1.23.2
      - name: step3
        type: apply-deployment
        properties:
          image: 10.113.xx.x:31104/library/nginx:v1.23.2
      - name: step-gg
        type: step-group
        subSteps:
          - name: apply-sub-step1
            type: apply-deployment
            properties:
              image: 10.113.xx.x:31104/library/nginx:v1.23.2
          - name: apply-sub-step2
            type: apply-deployment
            properties:
              image: nginx:v1.23.2

The following is the official source code call (workflow 0.6):

clientN := mgr.GetClient()

run := &wfv1alpha1.WorkflowRun{}
ctx := context.Background()
key := types.NamespacedName{Namespace: "default", Name: "dependency-ng"}
err = clientN.Get(ctx, key, run)
if err != nil {
    fmt.Println("err1")
    panic(err)
}

err = utils.SuspendWorkflow(ctx, clientN, run, "")

if err != nil {
    fmt.Println("err2")
    panic(err)
}

workflowrun cant not suspend

$ kubectl get workflowruns.core.oam.dev
dependency-ng   executing   23m

The following is my custom call based on the workflow source code of the previous version(workflow 0.4-):

	//Manually try to pause the task
	stepName := ""
	run.Status.Suspend = true
	steps := run.Status.Steps
	found := stepName == ""

	for i, step := range steps {
		if step.Phase == wfv1alpha1.WorkflowStepPhaseRunning {
			if stepName == "" {
				utils.OperateSteps(steps, i, -1, wfv1alpha1.WorkflowStepPhaseRunning)
			} else if stepName == step.Name {
				utils.OperateSteps(steps, i, -1, wfv1alpha1.WorkflowStepPhaseRunning)
				found = true
				break
			}
		}
		for j, sub := range step.SubStepsStatus {
			if sub.Phase == wfv1alpha1.WorkflowStepPhaseRunning {
				if stepName == "" {
					utils.OperateSteps(steps, i, j, wfv1alpha1.WorkflowStepPhaseRunning)
				} else if stepName == sub.Name {
					utils.OperateSteps(steps, i, j, wfv1alpha1.WorkflowStepPhaseRunning)
					found = true
					break
				}
			}
		}
	}
    if err := retry.RetryOnConflict(retry.DefaultBackoff, func() error {
		return clientN.Status().Patch(ctx, run, client.Merge)
	}); err != nil {
		fmt.Println(err)
	}

workflowrun Suspend work

$ kubectl get workflowruns.core.oam.dev
dependency-ng   suspending   23m

Expected behavior

Screenshots

Workflow Version

v0.6.0

Cluster information

Additional context

[Feature] Merge op.cue for kubevela and workflow

Is your feature request related to a problem? Please describe.

Merge op.cue for kubevela and workflow to make it consist.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

unable to update arrays via parameterKey

Describe the bug
When attempting to update an array in values, it's no longer possible to do so. Instead, it creates a map["0"] entry.

To Reproduce
use a parameterKey myscore[0]

Expected behavior

Expected it to produce values [99]

Instead it produces

{"0":99}

Screenshots
Actual
image
Expected
image

Additional context
I tracked down this drift from expected array output to the following PRs

After this PR, an oddity is emitted with [99,...] instead of [99] #86

Arrays can no longer be updated by workflows, instead get treated as maps
#112

[bug] restart a running workflow will cause workflow excutor always return Skipped

Describe the bug

restart a running workflow will cause workflow excutor always return Skipped

To Reproduce

  1. start a workflowrun
  2. use vela-cli to restart the workflowrun
    vela workflow restart xxx --step xxx
  3. workflow will not handled by controller, and the log like this
    "Skip this reconcile" workflowrun="default/test-xxx" spanID="i-dv1uiiok"

Expected behavior

Screenshots

Workflow Version
v0.6.0

Cluster information

Additional context

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.