step-security / harden-runner

Network egress filtering and runtime security for GitHub-hosted and self-hosted runners

Harden Github-hosted and self-hosted runners

Harden-Runner provides network egress filtering and runtime security for GitHub-hosted and self-hosted runners.

Learn how Harden-Runner works through the video below, which shows how it detected a supply chain attack on a Google open-source project.

3,000+ open source projects use Harden-Runner

CISA Explore	Microsoft Explore	Google Explore	DataDog Explore	Intel Explore	Kubernetes Explore	Node.js Explore	AWS Explore

Why

There are two main threats from compromised workflows, dependencies, and build tools in a CI/CD environment:

1. Exfiltration of CI/CD credentials and source code
2. Tampering of source code, dependencies, or artifacts during the build to inject a backdoor

Harden-Runner monitors process, file, and network activity to:

	Countermeasure	Prevent Security Breach
1.	Monitor and block outbound network traffic at the DNS, HTTPS (Layer 7), and network layers (Layers 3 and 4) to prevent exfiltration of code and CI/CD credentials	To prevent the Codecov breach scenario
2.	Detect if source code is being tampered during the build process to inject a backdoor	To detect the XZ Utils and SolarWinds incident scenario
3.	Detect poisoned workflows and compromised dependencies	To detect Dependency confusion and Malicious dependencies

How

GitHub-Hosted Runners

1. Add the step-security/harden-runner GitHub Action to your GitHub Actions workflow file as the first step in each job.

   steps:
     - uses: step-security/harden-runner@63c24ba6bd7ba022e95695ff85de572c04a18142 # v2.7.0
       with:
         egress-policy: audit

2. In the workflow logs and the job markdown summary, you will see a link to security insights and recommendations.

   

3. Click on the link (example link). You will see a process monitor view of network and file events correlated with each step of the job.

   

4. In the Recommended Policy tab, you'll find a recommended block policy based on outbound calls aggregated from the current and past runs of the job. You can update your workflow file with this policy, or alternatively, use the Policy Store to apply the policy without modifying the workflow file. From now on, any outbound calls not in the allowed list will be blocked.

   

Hands-On Tutorials

You can use GitHub Actions Goat to try Harden-Runner. You only need a GitHub Account and a web browser.

Hands-on Tutorials for GitHub Actions Runtime Security:

1. Filter Egress Network Traffic
2. Detect File Tampering

Support for Self-Hosted Runners and Private Repositories

Hardening for self-hosted runners and GitHub-hosted runners used in private repositories is supported with a commercial license. Check out the documentation for more details.

• To use Harden-Runner in a Private repository, you must install the StepSecurity Actions Security GitHub App.
• This is needed to access the GitHub Actions API and to authenticate users to access the insights URL for private repositories.
• If you use Harden-Runner GitHub Action in a private repository, the generated insights URL is NOT public. Only those who have access to the repository can view it.

Read this case study on how Kapiche uses Harden-Runner to improve software supply chain security in their private repositories.

Self-Hosted Actions Runner Controller (ARC) Runners

Explore demo workflows using self-hosted ARC Runner and ARC Harden-Runner here.

Actions Runner Controller (ARC) is a Kubernetes operator that orchestrates self-hosted runners for GitHub Actions.

• Instead of adding the Harden-Runner GitHub Action in each job, you'll need to install the ARC Harden-Runner daemonset on your Kubernetes cluster.
• Upon installation, the ARC Harden-Runner daemonset monitors all jobs run on the cluster; you do NOT need to add the Harden-Runner GitHub Action to each job for audit mode. You do need to add the Harden-Runner GitHub Action to jobs where you want to enable block mode.
• Please email [email protected] for instructions on how to install the ARC-Harden-Runner daemonset on your Kubernetes cluster.

Self-Hosted VM Runners (e.g. on EC2)

Explore demo workflows using self-hosted VM Runners and Harden-Runner here.

• Instead of adding the Harden-Runner GitHub Action in each job, you'll need to install the Harden-Runner agent on your runner image (e.g. AMI). This is typically done using packer or as a post-install step when using the https://github.com/philips-labs/terraform-aws-github-runner project to setup runners.
• The Harden-Runner agent monitors all jobs run on the VM, both ephemeral and persistent runners are supported; you do NOT need to add the Harden-Runner GitHub Action to each job for audit mode. You do need to add the Harden-Runner GitHub Action to jobs where you want to enable block mode.
• Please email [email protected] for instructions on how to install the Harden-Runner agent on your self-hosted VM runners. This agent is different than the one used for GitHub-hosted runners.

Features at a glance

For details, check out the documentation at https://docs.stepsecurity.io

📶 View outbound network traffic

Applies to both GitHub-hosted and self-hosted runners

Harden-Runner monitors all outbound traffic from each job at the DNS and network layers

• After the workflow completes, each outbound call is correlated with each step of the job, and shown in the insights page
• For self-hosted runners, no changes are needed to workflow files to monitor egress traffic
• A filtering (block) egress policy is suggested in the insights page based on the current and past job runs

🌐 View outbound HTTPS traffic

Applies to GitHub-hosted and self-hosted VM runners

Harden-Runner can monitor outbound HTTPS requests. This feature is supported with a commercial license.

• HTTPS events are monitored using eBPF (no MITM proxy is used)
• If a HTTP PUT/ POST/ PATCH call is made to github.com or api.github.com hosts to a HTTP Path with a different organization than where the workflow is running, the call is marked as anomalous
• As of now, only HTTPS calls to github.com and api.github.com hosts are monitoried.

⚠️ Detect anomalous outbound network traffic

Applies to both GitHub-hosted and self-hosted runners

Harden-Runner creates a baseline of outbound traffic for each job during the first few runs that it monitors

• After the baseline is created, any new outbound destinations are marked as anomalous in the insights page
• You can view the list of all anomalous outbound network traffic in the Runtime detections page on the dashboard

🚦 Filter outbound network traffic to allowed endpoints

Applies to both GitHub-hosted and self-hosted runners

Once allowed endpoints are set in the policy in the workflow file, or in the Policy Store

• Harden-Runner blocks egress traffic at the DNS (Layer 7) and network layers (Layers 3 and 4)
• It blocks DNS exfiltration, where attacker tries to send data out using DNS resolution
• Wildcard domains are supported, e.g. you can add *.data.mcr.microsoft.com:443 to the allowed list, and egress traffic will be allowed to eastus.data.mcr.microsoft.com:443 and westus.data.mcr.microsoft.com:443

📁 Detect tampering of source code during build

Applies to both GitHub-hosted and self-hosted runners

Harden-Runner monitors file writes and can detect if a file is overwritten.

• Source code overwrite is not expected in a release build
• All source code files are monitored, which means even changes to IaC files (Kubernetes manifest, Terraform) are detected
• You can enable notifications to get one-time alert when source code is overwritten
• For self-hosted runners, no changes are needed to workflow files for file monitoring

🚫 Run your job without sudo access

Applies to GitHub-hosted runners

GitHub-hosted runner uses passwordless sudo for running jobs.

• This means compromised build tools or dependencies can install attack tools
• If your job does not need sudo access, you see a policy recommendation to disable sudo in the insights page
• When you set disable-sudo to true, the job steps run without sudo access to the GitHub-hosted Ubuntu VM

🔔 Get security alerts

Applies to both GitHub-hosted and self-hosted runners

Install the StepSecurity Actions Security GitHub App to get security alerts.

• Email and Slack notifications are supported
• Notifications are sent when outbound traffic is blocked or source code is overwritten
• Notifications are not repeated for the same alert for a given workflow

Discussions

• If you have questions or ideas, please use discussions.
• For support for self-hosted runners and private repositories, email [email protected].
• If you use a different CI/CD Provider (e.g. Jenkins, Gitlab CI, etc), and would like to use Harden Runner in your environment, please email [email protected]

How does it work?

GitHub-Hosted Runners

For GitHub-hosted runners, Harden-Runner GitHub Action downloads and installs the StepSecurity Agent.

• The code to monitor file, process, and network activity is in the Agent.
• The agent is written in Go and is open source at https://github.com/step-security/agent
• The agent's build is reproducible. You can view the steps to reproduce the build here

Self-Hosted Actions Runner Controller (ARC) Runners

• ARC Harden Runner daemonset uses eBPF
• You can find more details in this blog post: https://www.stepsecurity.io/blog/introducing-harden-runner-for-kubernetes-based-self-hosted-actions-runners
• ARC Harden Runner is NOT open source.

Self-Hosted VM Runners (e.g. on EC2)

• For self-hosted VMs, you add the Harden-Runner agent into your runner image (e.g. AMI).
• Agent for self-hosted VMs is NOT open source.

Limitations

GitHub-Hosted Runners

1. Only Ubuntu VM is supported. Windows and MacOS GitHub-hosted runners are not supported. There is a discussion about that here.
2. Harden-Runner is not supported when job is run in a container as it needs sudo access on the Ubuntu VM to run. It can be used to monitor jobs that use containers to run steps. The limitation is if the entire job is run in a container. That is not common for GitHub Actions workflows, as most of them run directly on ubuntu-latest. Note: This is not a limitation for Self-Hosted runners.

Self-Hosted Actions Runner Controller (ARC) Runners

1. Since ARC Harden Runner uses eBPF, only Linux jobs are supported. Windows and MacOS jobs are not supported.

Self-Hosted VM Runners (e.g. on EC2)

1. Only Ubuntu VM is supported. Windows and MacOS jobs are not supported.

harden-runner's Issues

Make egress-policy block explicit on the insights website

Even though block is the default value for egress-policy, it is better to have it be set to block explicitly. This makes it easier to change it to audit if traffic is blocked, and the workflow needs to be run in audit mode again.

https://github.com/Automattic/vip-go-mu-plugins/pull/2815/files#r786925704

Pin release of harden-runner to agent

Linked issue: step-security/agent#50

Release latest version

Investigate run - insights not generated

https://app.stepsecurity.io/github/jauderho/dockerfiles/actions/runs/1742696205

https://app.stepsecurity.io/github/jauderho/dockerfiles/actions/runs/1747044469

Start link to insights on a new line in the logs

Does not work with some container images

  semgrep:
    permissions:
      contents: read # for actions/checkout to fetch code
    name: semgrep-scan
    runs-on: ubuntu-20.04
    container:
      image: returntocorp/semgrep
    if: (github.actor != 'dependabot[bot]')
    steps:
    - name: Harden Runner
      uses: step-security/harden-runner@9b0655f430fba8c7001d4e38f8d4306db5c6e0ab
      with:
        egress-policy: audit

    - uses: actions/checkout@3df53dd32d858478710a6127bcd8b9d8b7182e16
    - run: semgrep ci || true

This fails with the below error

/bin/sh: sudo: not found
Error: Command failed: sudo mkdir -p /home/agent
/bin/sh: sudo: not found

Add to the GitHub marketplace

Consider adding simple checksum validation for the downloaded tar prior to extraction / execution

harden-runner/src/setup.ts

Line 57 in 0045e63

"https://github.com/step-security/agent/releases/download/v0.8.6/agent_0.8.6_linux_amd64.tar.gz"

Generate dist folder using GitHub Actions

Show blocked traffic in annotations

Setup prod api domain

• Decide on beta vs only prod
• Add prod api into main branch

Investigate run

https://github.com/MTRNord/matrix-art/actions/runs/1743376122

Update readme to clarify experience for private repo

related issue: #130

Create PR into 3 repos to pilot

Add more scenarios to canary

• use of container
• longer run time

step-security/harden-runner stops npm install

In one of my github actions i setup 9b0655f # v1 with audit egress policy. I was using https://github.com/wallies/action-netlify-deploy to setup my netlify previews. The issue was when it came to install my dependencies via npm ci or npm install it would just hang. I removed step-security/harden-runner step and everything worked. Im using node lts 16.14

Make it configurable, if mode is block, to not send telemetry to API

When user sets this mode, e.g.

- name: Harden Runner
        uses: step-security/harden-runner@rc
        with:
          egress-policy: block
          disable-telemetry: true
          allowed-endpoints: 
            github.com:443

agent should not make calls to agent.api.stepsecurity.io.
Other allowed-endpoints can still be called.

• Do not show link - clarify that insights will not be generated
• Add this attribute to the config

Use env variable names for GitHub related endpoints

Use GitHub token to avoid rate limiting

A job failed because of this error. Initial investigation seems to suggest it is coming from tool-cache

https://github.com/nvm-sh/nvm/runs/4919179993?check_suite_focus=true#step:2:5

Trust for agent build

Hi, very exciting project!

I took a look at https://github.com/step-security/harden-runner/blob/main/src/checksum.ts#L13 and saw that you are verifying a checksum before installing the agent. How can I verify this checksum? I made a local build of the agent with goreleaser and got a different checksum for the generated .tar.gz.

Being able to verify the integrity of the agent binary feels like a critical step in improving the supply chain integrity.

Use latest/ supported version of node

#61 (comment)

Add service logs to post step

Protecting link in private repositories

This tool looks really great, keen to give it a try.

I'm thinking of using it in a private repo. It seems that if I do that then the generated URL will be public and it will follow the predictable pattern of {org}/{repo}/actions/runs/{runid}. Is that right?

I understand I can disable sending telemetry to the step security API which will ensure the details aren't made public - but in that case it seems there's no practical way of doing an initial workflow run to determine which endpoints etc. to whitelist (e.g. with egress-policy: audit)

• Is there any way to have the report that's hosted on app.stepsecurity.io generated as a static file and saved as a workflow artefact?
• Is there any way to have the report URL contain, say, a secret string? A random URL so it's not guessable, but is still publicly accessible?
• do you have other plans for handing this issue?

harden runner not loading security console

Demo Harden Runner workflow for nuget/setup-nuget

name: CI
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
    
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2
      - uses: step-security/harden-runner@rc
        with:
          egress-policy: audit
      - name: nuget/setup-nuget
        uses: nuget/setup-nuget@v1
        with:
          nuget-version: '5.x'

Generated Workflow for nuget/setup-nuget

Output Step Security Console from harden-runner
https://app.stepsecurity.io/github/Devils-Knight/Knowledge-base/actions/runs/1693963110

Add StepSecurity: Harden-Runner as prefix to code overwrite warning

It is hard to tell what is generating the annotation

e.g. https://github.com/ossf/scorecard/actions/runs/2018689479

Do not show annotation for api.snapcraft.io

api.snapcraft.io is called by snapd and this call is made from most jobs. Showing this in the annotations when this traffic is blocked causes unnecessary confusion. Do not show this in annotation, even when it is actually being blocked.

Create PR in pilot repos with changes

• #26
• #32
• #30

Bug: Output of harden-runner insight should use commit hash instead

I just testing and implementing harden-runner after starting with the scorecard action. Repo is here: https://github.com/jauderho/psfiles

So if the push is for actions to use commit hashes instead of version tags, the output page of the insights should utilize the commit hash instead.

For example, https://app.stepsecurity.io/github/jauderho/psfiles/actions/runs/1731266664

Recommendation is for

- uses: step-security/harden-runner@v1
  with:
    allowed-endpoints: 
      api.github.com:443
      github.com:443

Instead, it really should be (and changing as necessary as the action gets updated)

- uses: step-security/harden-runner@14dc64f30986eaa2ad2dddcec073f5aab18e5a24 # v1
  with:
    allowed-endpoints: 
      api.github.com:443
      github.com:443

Similarly, the README.md for this repo should indicate/recommend the use of hashes instead.

Investigate block-and-fail mode

Where job fails if there is call to endpoints not in the allowed list.

#61 (comment)

allow-endpoints should accept wildcards or better handle Github artifact hosts

As per the GitHub runner requirements, a GitHub runner that needs to handle caches and artefacts should be able to connect to *.blob.core.windows.net hosts.

E.g. my build workflow connected to kv4gacprodeus2file3.blob.core.windows.net:443, to download the yarn cache, and then optionally would upload it again in a post step (via action/setup-node action, which uses actions/cache).

The GH cache action uses an indirect API; an API call is made to artifactcache.actions.githubusercontent.com to get the blob location, which in my case was kv4gacprodeus2file3.blob.core.windows.net. However, there is no guarantee that a next run will use that same hostname!

My next run made a connection to gheus21ubt20eus26diag.blob.core.windows.net during the post-complete step, as the provisioner tried to upload some artefact. The harden-runner blocked this as the host was not whitelisted.

I fully realise that just whitelisting *.blob.core.windows.net is not a great idea either; there could be any number of malicious payloads lurking in Azure blob stores, so ideally there should be a way to whitelist just the known GitHub artefact and cache hosts. This may require petitioning GitHub to use dedicated hostnames for their cache / artefact hosts and / or the use of SSL certificates on those hosts that can be verified.

Copy block on the insights page should be the entire workflow

#61 (comment)

Formatting bug with latest harden runner ouput

When copying the latest code block output from https://app.stepsecurity.io/github/jauderho/dockerfiles/actions/runs/1800983360 , I notice an issue. There seems to be a spurious > in the code block in the same line as allowed-endpoints:.

See below for example,

- name: Harden Runner
  uses: step-security/harden-runner@14dc64f30986eaa2ad2dddcec073f5aab18e5a24 # v1
  with:
    egress-policy: block
    allowed-endpoints: >
      95s5acprodeus1file6.blob.core.windows.net:443
      api.github.com:443
      artifactcache.actions.githubusercontent.com:443
      crates.io:443
      dl-cdn.alpinelinux.org:443
      ghcr.io:443
      github.com:443
      production.cloudflare.docker.com:443
      registry-1.docker.io:443
      sh.rustup.rs:443
      static.crates.io:443
      static.rust-lang.org:443

Fails to run with kubernetes custom github-runners

Hi I wanted to switch over to using https://github.com/actions-runner-controller/actions-runner-controller/ for custom runners but it seems the default image of it doesn't set the "$USERS" variable.

This results in this error: https://github.com/MTRNord/matrix-art/runs/5368009721?check_suite_focus=true

Any ideas how I could fix this or what is expected?

Update readme

• Add links to more users
• Mention that one can add it using workflows and also fix token permissions
• Add insights link with each repo using it
• Add link to discussions
• Improve screen shots
• Add link to GitHub runner issue - since api.snapcraft.io comes up often
• Remove slack since discussions is being used more often
• Include info about private repos - preview
• Add testimonials?
• Add discussion on SLSA
• Add discussion on verifying tool checksums
• Yaml formatting

Add policy input

- uses: step-security/harden-runner@main
   with:
    allowed-endpoints:
      api.github.com:443

AND

- uses: step-security/harden-runner@main
   with:
    policy: audit

Investigate run

https://app.stepsecurity.io/github/Automattic/vip-go-mu-plugins/actions/runs/1757921355

Create a tag for releases

Update readme

allowed-endpoints should either be a list or use a block-style indicator

The allowed_endpoints block is, apparently, a blob of text. The recommendations output by the hardener look something like this:

- name: Harden Runner
  uses: step-security/harden-runner@14dc64f30986eaa2ad2dddcec073f5aab18e5a24 # v1
  with:
    egress-policy: block
    allowed-endpoints: 
      api.github.com:443
      artifactcache.actions.githubusercontent.com:443
      github.com:443
      kv4gacprodeus2file3.blob.core.windows.net:443

which my Visual Studio code editor then neatly reformats to:

  - name: Harden Runner
    uses: step-security/harden-runner@14dc64f30986eaa2ad2dddcec073f5aab18e5a24 # v1
    with:
      egress-policy: block
      allowed-endpoints: api.github.com:443
        artifactcache.actions.githubusercontent.com:443
        github.com:443
        kv4gacprodeus2file3.blob.core.windows.net:443

note that the api.github.com:443 line has shifted up.

If the allowed-endpoints format were to be a list, the above would instead look like

- name: Harden Runner
  uses: step-security/harden-runner@14dc64f30986eaa2ad2dddcec073f5aab18e5a24 # v1
  with:
    egress-policy: block
    allowed-endpoints: 
      - api.github.com:443
      - artifactcache.actions.githubusercontent.com:443
      - github.com:443
      - kv4gacprodeus2file3.blob.core.windows.net:443

or if a block-style indicator for multi-line text was used, such as >:

- name: Harden Runner
  uses: step-security/harden-runner@14dc64f30986eaa2ad2dddcec073f5aab18e5a24 # v1
  with:
    egress-policy: block
    allowed-endpoints: >
      api.github.com:443
      artifactcache.actions.githubusercontent.com:443
      github.com:443
      kv4gacprodeus2file3.blob.core.windows.net:443

Both formats are in canonical form and would not cause a reformat. I used the latter in my configurations.

Reduce timeout duration

Implement an egress-policy: warn-only mode

microsoft/msquic#2310 (comment)

It should not block traffic, but should warn if there are new endpoints.

Part of the insights URL is being masked out

https://github.com/jauderho/dockerfiles/runs/4929417948?check_suite_focus=true#step:3:9

#61 (comment)

Update description in package.json

Change it to Security agent for GitHub-hosted runner to monitor the build process

Use publish action

• Decide on release method
• Set a version for first release

Add annotation for link to security insights

Call monitor run API

Address codeql issues

egress-policy: audit Is Blocking Outbound UDP 443

I am having to remove this from any of my actions that makes outbound UDP 443 (QUIC) connections. When this is enabled I get no connectivity. Related: microsoft/msquic#2534 and nibanks/msh3#28.

Update naming for outbound endpoints

• should allow for wildcards

Update readme

Recommend use of tag instead of branch