stackstorm-exchange / ci Goto Github PK

Please consider appending https://www.gitignore.io/api/pycharm to the file. Also, keep an eye for toptal/gitignore.io#186 (comment) due to controversial nature of the the problem.

If you are using the st2.conf setting:

[packs]
enable_common_libs = True

Then circle tests fail for actions since the python path is:

Running pylint on pack: repo
PYTHONPATH=/tmp/st2/st2actions:/tmp/st2/st2api:/tmp/st2/st2auth:/tmp/st2/st2client:/tmp/st2/st2common:/tmp/st2/st2debug:/tmp/st2/st2exporter:/tmp/st2/st2reactor:/tmp/st2/st2stream:/tmp/st2/st2tests:/home/circleci/repo/actions/lib:/tmp/st2/st2common:
PYTHON_BINARY=/home/circleci/virtualenv/bin/python
Using config file /home/circleci/ci/lint-configs/python/.pylintrc-exchange

I think the solution is to be able to pass an envvar to the build job for a pack that detects that it is using common libs then loads an st2_test.conf file that sets the enable_common_libs option

Follow up to #104.

When the index.py script does not have the correct credentials, it does not properly detect this and proceeds to error out later on when it attempts to extract information from the response:

+ /home/circleci/virtualenv/bin/python /home/circleci/ci/.circle/index.py --glob '~/index/v1/packs/*/pack.yaml' --output '~/index/v1/'
gh: Bad credentials (HTTP 401)
Traceback (most recent call last):
  File "/home/circleci/ci/.circle/index.py", line 217, in <module>
    get_available_versions()
  File "/home/circleci/ci/.circle/index.py", line 166, in get_available_versions
    repos = page["data"]["repositoryOwner"]["repositories"]["nodes"]
KeyError: 'data'

The script should better detect when it doesn't have the appropriate credentials and give a more user friendly error message about what the underlying issue likely is, and most importantly what the user can do to fix it (eg: go regenerate the PAT for the pack, then set the MACHINE_PASSWORD environment variable in the CircleCI project settings, and rerun the CircleCI job).

Hopefully the PATs will expire less often now that we "use" them in the weekend CI runs, but that means that we will not be as familiar with those errors when they do arise since they will happen less frequently, so detecting errors and being more explicit about the exact problem and the way to fix it will help point us future maintainers to the solution in the future.

@Kami do we still need this line in our Makefile?

https://github.com/StackStorm-Exchange/ci/blob/master/.circle/Makefile#L123-L124

	# Note: Line below is a work around for corrupted Travis Python wheel cache issue
	. $(VIRTUALENV_DIR)/bin/activate && $(VIRTUALENV_DIR)/bin/pip install --no-cache-dir --no-binary :all: --upgrade --force-reinstall greenlet

Looks like it came from here: StackStorm/st2contrib@11ddb4a

Right now it looks like that all the pack repositories doesn't have Github repository description set.

I thought exchange bootstrap did that, but maybe I was wrong.

In any case, we should add a small script which takes get token and then sets Github repo description so it matches description field from pack.yaml metadata file.

After the script is available, we should run it on all the existing pack repos to ensure each repo has a description set.

Having a description set should help with Github and external search and SEO.

Over the last several weeks we've noticed and had to correct several packs where the tagged release version did not include the changes that were merged with the PR.

Looking into this i believe i found the culprit: https://github.com/StackStorm-Exchange/ci/blob/master/.circle/deployment#L33-L52

This code is where we create release tags on the pack repos. If you notice we're pulling a revlist for pack.yaml. What is happening is that changes made to a PR that are committed AFTER a pack.yaml has been updated are being excluded from the release.

Example: https://github.com/StackStorm-Exchange/stackstorm-terraform/pull/26/commits

You can see there are three commits in this PR:

• 1edf85c some code updates
• 8fa2dab update to pack.yaml
• 0313c69 merge from master -> this branch

Since 8fa2dab is the change that updated pack.yaml it is the one that got the 0.5.0 tag: https://github.com/StackStorm-Exchange/stackstorm-terraform/releases/tag/v0.5.0

This left out 0313c69 causing changes that were already merged into master from being excluded from the release.

A new release had to be cut in order to tag the latest commit.

StackStorm/st2#5564 added Changelog checker for stackstorm/st2 based on https://github.com/marketplace/actions/changelog-checker which works for PR forks.

90% of the PR reviews in Exchange were about asking to add the missing changelog and version.
We'll need GH Actions automation for StackStorm-Exchange notifying users to update 2 things via PR checks/messages:

• Changelog.md
• Version in pack.yaml

In this script, we try to retrieve the full list of repos by appending ?per_page=1000

https://github.com/StackStorm-Exchange/ci/blob/master/tools/common.sh#L14

This won't work - the maximum returned is 100. We need to traverse pages
https://developer.github.com/guides/traversing-with-pagination/

Summary

We maintain our own semi-fork of this repo at https://github.com/EncoreTechnologies/ci-stackstorm
The reason for this is because this repo isn't great at running from a developers laptop. Specifically it creates directories the ~/ci and ~/virtualenv, along with several other problems (outlined below).

Goals

• I would like to be able to utilize this repo for testing, at some level, so that our code always is testing against the upstream configs
• I would like this repo to be able to be utilized on a developers laptop
• I would like this repo to easily be able to test against multiple packs on the same machine
• I would like this repo to have support for both python2 and python3
• I would like this repo to have support for both Ubuntu and CentOS

Current problems/challenges/etc

Below is a list of things we had to change from these Makefiles and scripts to get the testing to work on a developers laptop:

CI repo

https://github.com/StackStorm-Exchange/ci/blob/master/.circleci/config.yml explicitly sets up the CI repo in ~/ci and and virtualenv in ~/virtualenv. This leads to potential issues when multiple
packs are tested on a single system, specifically around the ~/virtualenv. We've had cases where a user forgot to add in a dependency into requirements.txt and because the virtualenv was already setup in ~/virtualenv from another packs' run, we never noticed the problem until we had deployed the pack to our staging environment.

Proposed Changes

• Setup a Makefile that is designed to be placed in a Pack's root directory. This Makefile clones the ci repo and then executes the tests. This Makefile is designed to be kicked off by a user and can coordinate all steps needed to happen in order to run tests. Basically, merge functionality from .circleci/config.yml, .circle/dependencies, .circle/test together into the Makefile.
• Clone the CI repo into ./ci inside the pack's directory instead of ~/ci. Cloning locally means each pack has their own copy when testing against multiple packs on a single machine (preventing file corruption issues).

Dependencies

The dependency script that is run by this CI repo has a couple problems:

• It is not idempotent, so when a developer wants to run their tests a 2nd time, the dependency script errors, specifically around the git clone methods.
• The script hard codes the CI_DIR to be /home/circleci/ci
• The script expects a Debian instance and calls apt-get
• The script creates a ~/virtualenv

Proposed Changes

• Put these dependency resolution steps into a Makefile and have idempotent checks like:

	if [ ! -d "$(ST2_REPO_PATH)" ]; then \
		git clone https://github.com/StackStorm/st2.git --depth 1 --single-branch --branch $(ST2_REPO_BRANCH) $(ST2_REPO_PATH); \
	else \
		cd "$(ST2_REPO_PATH)"; \
		git pull; \
	fi;

• Put the dependencies into ./ci inside the pack's directory instead of ~/ci or /home/circleci/ci
• Put virtualenv into ./ci/virtualenv inside the pack's directory instead of ~/virtualenv
• Don't run any apt-get or yum commands.

packs-resource-register requires MongoDB

Currently the packs-resource-register test requires that a MongoDB instance be running. Unless the user has a docker up and a MongoDB container running, this test is hard to run.

Proposed Changes

• See if we can modify the test to not require MongoDB at all and just validate against known schemas?

StackStorm/community#7

Is our CI working properly for our Exchange packs?

I would have expected this PR here StackStorm-Exchange/stackstorm-vadc#1 to show results of checks. Looks like checks aren't being run?

StackStorm/community#26

When pushing changes to the index repo, the push step sometimes fails due to a race condition, or other situations such as invalid access token.

This causes the deployment step to exit, but the logs are unhelpful, because the code contains this:

git -C ~/index push origin 2>/dev/null

It would be helpful to have better logging here to identify the cause of the failure. (as well as fix the race condition, see #65)

This PR #36 causes problems when setting up a new repo.

It attempts to set the repo based upon the contents of pack.yaml. But for newly created repos, there is no pack.yaml yet. So it fails to download pack.yaml, and the script exits.

The problem when there were no commits in the repository for 60 days, - Github Automatically disables CI for it.
It happens all the time when we review new contributions in Exchange and figure out after merging that CI didn't run resulting in post-merge linting errors.

Possible solutions:

1. https://github.com/marketplace/actions/keepalive-workflow heart beat commits .. 🤢 sounds like a hack

2. Per @nzlosh There's a REST API endpoint that can be called to enable workflows, so that'd be more robust. https://docs.github.com/en/rest/actions/workflows?apiVersion=2022-11-28#enable-a-workflow We could schedule an St2 workflow that iterates all exchange repos and calls the API to enable CI workflows. From the possible workarounds, this looks like the cleanest and most maintainable way to workaround the github limitation.

3. Considering the StackStorm instance is automatically started/stopped, 3rd option could be an AWS Lambda function trying to re-enable GH Action Workflows for the Exchange repos. Example solution https://stackoverflow.com/a/68591566/4533625

Hello,

It seems that since commits like this one were pushed around August 5th, the CI has been broken.

It fails during the flake8 step with the following error:

Force flake8 checks on all files
Makefile:102: *** Recursive variable 'PYTHONPATH' references itself (eventually).  Stop.

Exited with code exit status 2

I first noticed it breaking today while working on a new feature branch so I relaunched on old CircleCI pipeline that was working on August 3rd and it failed with the same error.

When Using the CircleCI Local CLI for a StackStorm Exchange pack using the following command:

circleci local execute --job build_and_test

the build pauses to confirm the authenticity of host github.com and then stalls due to the lack of access to an ssh public key:

====>> Download dependencies
  #!/bin/bash -eo pipefail
git clone -b master [email protected]:stackstorm-exchange/ci.git ~/ci
~/ci/.circle/dependencies

Cloning into '/home/circleci/ci'...
The authenticity of host 'github.com (192.30.253.112)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)? yes

This is due to the following git checkout using ssh and not the git protocol:

I'll raise a PR to change this over and close this issue.

We're seeing 500 responses with the Slack webhook integration for Exchange packs - e.g.: https://github.com/StackStorm-Exchange/stackstorm-vault/settings/hooks/10916336#delivery-response

Headers:

Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 39
Content-Security-Policy: referrer no-referrer;
Content-Type: text/html
Date: Fri, 09 Dec 2016 13:55:41 GMT
Server: Apache
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Vary: Accept-Encoding
Via: 1.1 9552af4ceb2e0dd55964a80d1e549fd5.cloudfront.net (CloudFront)
X-Amz-Cf-Id: HZX-n7VkLLdLErEvYjTlGi52GafxpT5bZUG6nq0knQ9V7PUs565VzQ==
X-Cache: Error from cloudfront
X-Frame-Options: SAMEORIGIN
X-Slack-Backend: h

Body:

missing_text_or_fallback_or_attachments

Looks like something not working properly?

Interestingly, in the request, we see this content-type: application/x-www-form-urlencoded

But in the webhook configuration, it says use application/json
Unclear why.

Need to migrate to CircleCI 2.0 before August 2018

As decided in StackStorm/community#7, we should always check all files in all CircleCI workflows.

In https://github.com/StackStorm-Exchange/ci/blob/master/.circle/deployment#L23, the deployment script clones the github.com/StackStorm-Exchange/index repo, then updates index.json and the pack icon.png.

It then commits & pushes those changes.

If multiple packs are attempting to deploy changes around the same time, this can result in a race where multiple packs are attempting to push to master at the same time.

Code needs to be updated to handle this, and attempt to rebase with latest changes, and push again. Possibly with a few retries.