circleci-public / continuation-orb Goto Github PK

Is there an existing issue that is already proposing this?

• I have searched the existing issues

Describe the problem imposed by not having this feature

What changed from 0.5.0 to 1.0.0 ?

Describe the solution you'd like

keep a changelog with breaking changes

Other

No response

Description

This continuation orb added the long-awaited support in monorepos where we have first step to evaluate "what to do", and the continue step to actually do it.

The next step to make this solution monorepos friendly is to allow nice integration with github "required" status so PRs would get green-go only if: (a) no workflows actually ran or (b) all workflows that ran actually succeeded.

The idea is that all statuses would be aggregated into a single status context in Github (since Github only allows to pre-configure the required context).

The easiest way to implement it first is to have the continue job configured to verify that all scheduled workflows actually passed and make the continuation workflow status the required one.

Examples:

1. Dana opens a PR with changes to foo area, which is dynamically translated into backend-build workflow. Her PR required check is backend-build
2. Bekah opens a PR with changes to bar area, which is dynamically translated into frontend-build workflow. Her PR required check is frontend-build
3. John opens a PR to protos area, which is dyanmically translated into protobuf-build, backend-build and frontend-build workflows. His PR required checks are all three workflows.

We can extend the continue job with the following syntax:

    jobs:
      - continuation/continue:
          configuration_path: ./cont-config.yml
          propagate: true  # waits for downstream builds and propagate their aggregated statuses

Another advanced option:

    jobs:
      - continuation/continue:
          configuration_path: ./cont-config.yml
          propagate_name_patterns: ["required-.*"]  # waits for downstream workflows with name that matches one of the given patterns, and propagate their aggregated statuses

Supporting Documentation Links:

https://docs.github.com/en/github/administering-a-repository/defining-the-mergeability-of-pull-requests/about-protected-branches#require-status-checks-before-merging

Orb version:

0.2.0

What happened:

When invoking the command continue on an alpine instance it fail with /bin/sh: syntax error: unexpected redirection

Expected behavior:

Working in the same way as on any other linux distribution that use bash.

Additional Information:

The issue is because of the syntax used in the continue.sh script.

I'm going to open a PR to fix the issue

Orb version: 0.2.0

What happened:

When writing shell scripts directly within the command of a step within a generated script is does not work the same as it does in the parent. For example in the parent config.yml, I have a step when a set a variable and then echo it. The value of the variable is displayed. This does not happen within a generated config. I have linked below an example repo showcasing this.

Expected behavior:

Expect the same functionality that commands allow in parent configs to work in generated configs

Additional Information:

Repo with an example of the issue: https://github.com/BECkvalentine/circle-ci-command-example/tree/main

Orb version:

0.2.0

What happened:

We're trying to use this in our self hosted cci setup (3.3.1), however when we attempt to use

circleci_domain: circleci.ourdomain.com, we get the following error.

Error calling workflow: 'setup'
Error calling job: 'setup'
Error calling command: 'continuation/continue'
Unexpected argument(s): circleci_domain

Expected behavior:

It should find that parameter and use it.

Describe Request:

I am generating a new CircleCI configuration file to give to this orb:

      - continuation/continue:
          configuration_path: ".circleci/continue_config.yml"

Unfortunately, sometimes there are simply no workflows that I'd want to continue into. For example, a PR that only updates README.md will not match any entries in my mapping for path-filtering-orb. In this case CircleCI considers the continued build an error because there is nothing to do:

In this case the documentation says to close out:

      - continuation/finish

These are two different steps - I'm not aware of how to conditionally call steps based on the output of a previous step!

It seems as though this orb should have a third command that can dynamically switch between 'continue' or 'finish' depending on a parameter.

Examples:

Something like this could probably work, if nothing else:

      - run:
          command: echo 'export CONTINUATION_ACTION=continue' >> $BASH_ENV
      - continuation/decide:
          configuration_path: ".circleci/continue_config.yml"
          action: $CONTINUATION_ACTION

I could also imagine there being a special check on the config on disk, for example if .circleci/continue_config.yml is empty (zero bytes) then finish instead of continuing.

Supporting Documentation Links:

• https://github.com/CircleCI-Public/api-preview-docs/blob/path-filtering/docs/setup-workflows.md

Executors are useful for parameterized execution environments that will be reused. As this executor is used exclusively within the continue job, it should not be necessary and can sometimes be confusing for users.

Instead, I would recommend setting the docker executor and image statically on the continue job, including the same tag parameter for the image version tag.

Describe Request:

This orb currently contains a mandatory checkout step.

If the repository had already been checked out by a previous job, then this step is redundant.

I believe the orb's configuration should allow to skip this checkout step.

For consistency, you could also consider adding an optional workspace_path parameter to attach a workspace, similarly to how it's done in the path-filtering-orb.

Thank you.

Describe Request:

I would like to be able to optionally use the continuation orb with when: on_fail.

The use case I have in mind is to kick off a manual approval override in the case of a failed job:

workflows:
  approve_manual_override:
    jobs:
      - manual-step-if-jobs-failed:
          type: approval

and/or kick off different downstream workflows depending if a particular job succeeded or failed.

I have this working in my codebase by copying the source of the orb as a command in my config.yml but would love for this to be incorporated into the upstream orb!

Examples:

New parameter:

      when:
        description: When to run the workflow ['on_fail', 'on_success']
        type: string
        default: 'on_success'

Usage:

    steps:
      - run:
          when: << parameters.when >>

Supporting Documentation Links:

Describe Request:

Currently few examples are provided and there is no example of how to use the parameters parameter. As the text states "can either be a JSON object containing parameters or file name" many users are likely to end up being tripped up as it is not an object you define, but rather a JSON string containing a object.

Examples:

not

   {"name":"John", "age":30, "car":null}

but instead

   '{"name":"John", "age":30, "car":null}'

Supporting Documentation Links:

Describe Requests

The continuation orb is not the only CircleCI API consumer. We have some jobs that monitor workflows, workarounds for mutex and other jobs that also perform API requests.

We noticed that on peak times the API calls may fail with error 429 (Too many request).
CircleCI support suggested to read the HEADER

We implemented the retry with backoff on our scripts but we rather that the official continuation orb would take care of that edge case on the official orb.

Thanks!

Is there an existing issue that is already proposing this?

• I have searched the existing issues

Describe the problem imposed by not having this feature

The checkout of the repo is taking too long while we need only one file

Describe the solution you'd like

Add a step to clone the repo shallow, or just the file

Other

No response

Orb version:

0.2.0

What happened:

No actual workflow was triggered

Expected behavior:

We are expecting to see the workflow defined in build.yml running.

Additional Information:

A tag was pushed to our GitHub repo and triggered CircleCI setup workflow.
At the end of the setup, we have this step:

- continuation/continue:
       configuration_path: .circleci/build.yml
       parameters: /tmp/pipeline-parameters.json

and the actual runtime output was:

{
  "continuation-key": "*********",
  "configuration": "< the content of our build.yml >",
  "parameters": < the content of our pipeline-parameters.json >
}
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21892  100    16  100 21876    166   222k --:--:-- --:--:-- --:--:--  222k
{"message":"OK"}
CircleCI received exit code 0

This should switch from my 2nd party badge service to the official one.

Currently, when a user tries to rerun a pipeline and the continuation key has expired, they get an unhelpful error message.

{"message":"Key has expired."}

Exited with code exit status 1
CircleCI received exit code 1

If you don't know the details of how dynamic configuration pipelines work, this message is meaningless. Could you update the API response to include a more helpful message or have the orb check the message and add more context/help?

Orb version:

0.2.0

What happened:

I have a monorepo with the following config:

version: 2.1

setup: true

orbs:
  continuation: circleci/[email protected]

jobs:
  setup:
    executor: continuation/default
    steps:
      - checkout
      - run:
          name: Determine the jobs to run
          command: OUTPUT_PATH=/tmp/workspace .circleci/scripts/diff.sh
      - persist_to_workspace:
          root: /tmp/workspace
          paths:
            - pipeline-parameters.json

  service1:
    executor: continuation/default
    steps:
      - checkout
      - attach_workspace:
          at: /tmp/workspace
      - continuation/continue:
          configuration_path: .circleci/service1.yml
          parameters: /tmp/workspace/pipeline-parameters.json

  service2:
    executor: continuation/default
    steps:
      - checkout
      - attach_workspace:
          at: /tmp/workspace
      - continuation/continue:
          configuration_path: .circleci/service2.yml
          parameters: /tmp/workspace/pipeline-parameters.json

workflows:
  build:
    jobs:
      - setup
      - service1:
          requires:
            - setup
      - service2:
          requires:
            - setup

setup generates parameters to run on each of the two separate config.

Running service1 triggers a new pipeline as expected. Running service2 fails the continuation with {"message":"Pipeline is not in setup state."}

Expected behavior:

Starting a new separate pipeline for each service being continued.

Additional Information:

As an example, in my current scenario .circleci/service1.yml config file would be:

version: 2.1

parameters:
  run-service1-build:
    type: boolean
    default: false
  run-service2-build:
    type: boolean
    default: false

jobs:
  test:
    # ...
  lint:
    # ...

workflows:
  test_and_lint:
    when: << pipeline.parameters.run-service1-build >>
    jobs:
      - lint
      - test

Orb version:

0.2.0

What happened:

using the example (updating version to latest orb version) from https://circleci.com/developer/orbs/orb/circleci/continuation:

version: '2.1'
orbs:
  continuation: circleci/[email protected]
workflows:
  use-my-orb:
    jobs:
      - continuation/continue:
          configuration_path: ./config.yml

results in message:

CIRCLE_CONTINUATION_KEY is required. Make sure setup workflows are enabled.

Expected behavior:

no errors

Additional Information:

Example should include setup: true

Describe Request:

On behalf of the team @bionicstork:

When using dynamic configuration and the orb "circleci/continuation" the created config file has a size limit since the orb is passing the config using command line arguments and not files.

We fixed this by copying your code and changing it so it will split the config file, stringify the yaml and join all the parts back together.

Examples:

CONFIG_PATH=<<parameters.configuration_path>>
PARAMETERS=<<parameters.parameters>>

if [ -z "${CIRCLE_CONTINUATION_KEY}" ]; then
echo "CIRCLE_CONTINUATION_KEY is required. Make sure setup workflows are enabled."
exit 1
fi

if [ -z "${CONFIG_PATH}" ]; then
echo "CONFIG_PATH is required."
exit 1
fi

PARAMS=$([ -f "$PARAMETERS" ] && cat "$PARAMETERS" || echo "$PARAMETERS")

if ! jq . >/dev/null 2>&1 \<<<"$PARAMS"; then
echo "PARAMETERS aren't valid json"
exit 1
fi

echo "{\"config\":\"" > final
split -l 1000 "$CONFIG_PATH" part

parts=$(ls | grep part)

for p in $parts
do
content=$(cat $p)
jq -n --stream \
--arg config "$content" \
'{"part": $config}' > jsoned

partStr=""
split -l 1 jsoned line
line=$(cat lineab)
line=${line:11}
line=$(echo "$line" | sed 's/.$//')

current=$(cat final)
echo "${current}\n${line}" > final
done

current=$(cat final)
echo "${current}\"}" > final

JSON_BODY=$( jq -n --stream \
--arg continuation "$CIRCLE_CONTINUATION_KEY" \
--slurpfile config final \
--arg params "$PARAMS" \
'{"continuation-key": $continuation, "configuration": $config[0].config, parameters: $params|fromjson}'
)

echo "$JSON_BODY" > json_body

cat json_body

[[ $(curl \
-o /dev/stderr \
-w '%{http_code}' \
-XPOST \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
--data @json_body \
"https://circleci.com/api/v2/pipeline/continue") \
-eq 200 ]]

for the https://github.com/CircleCI-Public/continuation-orb/blob/master/src/scripts/continue.sh script.

Supporting Documentation Links: