spinnaker / dcd-spec

Declarative Pipeline specification for Spinnaker

dcd-spec

A living document of the Spinnaker Pipeline Templates and greater Declarative Continuous Delivery config language, a way to store your app, pipelines and infrastructure as code within Spinnaker.

This repository is organized by major DCD features.

• Pipeline Templates Spec

Other resources:

• pipeline-templates - Public repo of Pipeline Templates for examples and sharing.
• spin-dcd-converter - Tool to bootstrap converting existing Pipelines to Pipeline Template YAML.
• pipeline2dcd - Web browser extension to convert JSON Spinnaker pipelines to DCD pipelines.

dcd-spec's Issues

Support remote references for modules & partials

It would be valuable to be able to publish & reference remote modules & partials within templates and configurations. This could either be explicitly declared as part of #4, then given aliases for reference, or just directly referenced at call-site.

# Template snippet
id: myExampleTemplate
metadata:
  name: myExampleTemplate
  description: Modules example template
  owner: [email protected]
  scopes: [global]
variables:
- name: regions
  description: A list of AWS regions to deploy into
  type: list
stages:
- id: deploy
  type: deploy
  config:
    clusters: |
      {% for region in regions %}
      - {% module spinnaker://deployClusterAws region=region %}
      {% endfor %}

# Stored totally separately in a different object.
modules:
- id: deployClusterAws
  usage: Defines a deploy stage cluster using the AWS cloud provider
  variables:
  - name: region
    description: The AWS region to deploy into
  when: 
  - "{{ region != 'ap-northeast-1' }}"
  definition:
    provider: aws
    account: myAccount
    region: "{{ region }}"

pipeline can not find any stages source with pipeline templates

while a support to pipeline TestA and TesB, the TestB is configuration a auto Trigger job depends TestA, And the TestB pipeline is config with pipeline templates from remote source, while i work TestB pipeline with single , it can be worked ok, but while i work TestA first, the TestB work by trigger, it can not work ok error by can not find any stages

Modules - When conditions are not working

Stage Definition

variables:
- name: secrets
  type: list
  defaultValue:
  - name: NA
    envVars:
    - NA
- id: deployToDev
  type: deploy
  name: Deploy to Dev
  config:
    clusters:
    - account: dcomdev03
      application: "test"
      cloudProvider: kubernetes
      containers:
      - envVars: |
          {% for secret in secrets %}
          {% for envVar in secret.envVars %}
          - {% module secretsEnvSourceModule envName=envVar secretName=secret.name %}
          {% endfor %}
          {% endfor %}

Module Definition

modules:
- id: secretsEnvSourceModule
  usage: Defines the env source for exposing secrets as env variables
  variables:
  - name: envName
    description: env variable name
  - name: secretName
    description: Secret Name
  when: 
  - "{{ secretName != 'NA' }}"
  definition:
    envSource:
      secretSource:
        secretName: "{{ secretName }}"
        key: "{{ envName }}"
    name: "{{ envName }}"

Pipeline

spin cli

Can you use spin cli for adding pipeline templates to spinnaker. This would helm the integration and to pull pipeline from the code

Support remote configuration definition

Some users want the ability to retrieve pipeline configurations from a remote source, similar to how we resolve pipeline templates from remote sources. This remote source should be capable of supporting different formats, such as JSON, Java Properties, etc.

Pipeline template not rendering correctly in 1.11.7

We are using pipeline templates to update/re-create our deployment pipelines on Spinnaker. At the moment, we are experiencing a very strange issue. Since the upgrade to 1.11.7, our templates relying on helm charts are not rendering pipelines correctly.
The helm release pipelines are composed of two stages:

1. bake, where we render the manifests using Helm2 then produce a base64 artifacts.
2. deploy, where the base64 artifact is deployed to k8s inside a dedicated namespace

The issue we are experimenting is with the bake stage, even though we are specifying an expected artifact in the template artifact. The artifact is not displayed in the deck ui.

And if the pipeline executes, it will fail with the following error:

{"exception":{"details":{"error":"Unexpected Task Failure","errors":["01D1XY4MG9NRMXJX2BADSJCM2Q: Input artifact 5384bf5f-4638-4bae-afbf-ba22f2bed72b could not be found in the execution"],"stackTrace":"java.lang.IllegalArgumentException: 01D1XY4MG9NRMXJX2BADSJCM2Q: Input artifact 5384bf5f-4638-4bae-afbf-ba22f2bed72b could not be found in the execution\n\tat com.netflix.spinnaker.orca.bakery.tasks.manifests.CreateBakeManifestTask.lambda$execute$0(CreateBakeManifestTask.java:81)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)\n\tat java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)\n\tat java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)\n\tat java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)\n\tat java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)\n\tat com.netflix.spinnaker.orca.bakery.tasks.manifests.CreateBakeManifestTask.execute(CreateBakeManifestTask.java:85)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$2$1.invoke(RunTaskHandler.kt:100)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$2$1.invoke(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withLoggingContext(RunTaskHandler.kt:311)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.access$withLoggingContext(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$2.invoke(RunTaskHandler.kt:99)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$2.invoke(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.AuthenticationAware$sam$java_util_concurrent_Callable$0.call(AuthenticationAware.kt)\n\tat com.netflix.spinnaker.security.AuthenticatedRequest.lambda$propagate$0(AuthenticatedRequest.java:97)\n\tat com.netflix.spinnaker.orca.q.handler.AuthenticationAware$DefaultImpls.withAuth(AuthenticationAware.kt:49)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withAuth(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1.invoke(RunTaskHandler.kt:98)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1.invoke(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:182)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:49)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:33)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:59)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:33)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withExecution(OrcaMessageHandler.kt:68)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withExecution(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withStage(OrcaMessageHandler.kt:55)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withStage(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withTask(OrcaMessageHandler.kt:42)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:175)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:69)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.q.MessageHandler$DefaultImpls.invoke(MessageHandler.kt:36)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.invoke(OrcaMessageHandler.kt)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.audit.ExecutionTrackingMessageHandlerPostProcessor$ExecutionTrackingMessageHandlerProxy.invoke(ExecutionTrackingMessageHandlerPostProcessor.kt:47)\n\tat com.netflix.spinnaker.q.QueueProcessor$pollOnce$1$1.run(QueueProcessor.kt:82)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n"},"exceptionType":"IllegalArgumentException","operation":"createBake","shouldRetry":false,"timestamp":1548266656338}

We first thought it was related to spinnaker/spinnaker#3737 (comment) but it seems to be a different issue. Is there any workaround we could apply to temporary solve that issue ?

Support deployment models

Placeholder.

Add the concept of deployment models to DCD spec. There's been requests for constants that could be used across templates. The easiest example being standardized AWS cluster configurations.

This would likely depend on #4 to see the full extent of its usefulness.

How to enable `pipelineTemplates` into `settings.js` of Deck?

Hi,

I am currently interesting by the Pipeline Templates feature of Spinnaker, for enforcing Infrastructure As Code in my deployments. So, I would like to enable this feature into my current Spinnaker deployment (deployed into Kubernetes).

I am reading this document: https://github.com/spinnaker/dcd-spec/blob/master/ALPHA-GETTING-STARTED.md

As I know how to configure orca-local.yml via my hal settings, I don't know how to configure Deck, by adding pipelineTemplates parameter into the settings.js (which is part of a "secret" in my Kubernetes deployment).

How do you do that?

Thanks

Thomas.