anchore / ci-tools

Contains scripts for running anchore engine in CI pipelines

Warning: The Anchore Inline Scan script is deprecated and will reach EOL on Jan 10, 2022. Please update your integrations to use Grype for CI-based vulnerability scanning or Syft. We will be updating our integrations that use inline_scan during that time to use Grype directly. Until that time all integrations will continue to function and get updated vulnerability data.

Until Jan 10, 2022: we will continue building inline_scan images based on v0.10.x of Anchore Engine and they will be updated daily for latest feed data. On Jan 10, 2022, we will stop building new versions of the images with updated vulnerability data and the data will be stale.

After Jan 10, 2022: users should be transitioned to Grype or Grype-based integrations.

Anchore CI tools

An assortment of scripts & tools for integrating Anchore Engine into your CI/CD pipeline.

scripts/inline_scan

• Allows scanning and analysis of local docker images.
• Invokes the anchore/inline-scan container to perform a vulnerability scan or image analysis.

scripts/anchore-ci-tools.py

• Currently only supports docker based CI/CD tools.
• Script is intended to run directly on Anchore Engine containers.
• Used by scripts/inline_scan & should only be used when inline_scan is not an option.

scripts/build.sh

• Used to build any version of the inline_scan container.
• Also can be used to run CI tests locally.

CircleCI Orbs

Source code for CircleCi orbs has been moved to a new GitHub Repository.

Anchore inline-scan container

Image is built using the official Anchore Engine image base. It contains a Postgresql database pre-loaded with Anchore vulnerability data from https://github.com/anchore/engine-db-preload daily. Also contains a Docker registry which is used for passing images to Anchore Engine for vulnerability scanning.

Container is built using the scripts/build.sh script. The version of anchore-engine to build with can be specified with the environment variable ANCHORE_VERSION=<VERSION>.

After building the inline_scan container locally, the scripts/inline_scan script can be called using this container by setting the environment variable ANCHORE_CI_IMAGE=stateless_anchore:ci.

Inline vulnerability scanner

Wrapper script for inline_scan container, requires Docker & BASH to be installed on system running the script.

• Call script directly from github with:

  curl -s https://ci-tools.anchore.io/inline_scan-v0.10.0 | bash -s -- [OPTIONS] <IMAGE_NAME>

Pull multiple images from dockerhub, scan and generate reports.

inline_scan -p -r alpine:latest ubuntu:latest centos:latest

Pass Dockerfile to image scan, after docker build.

docker build -t example-image:latest -f ./Dockerfile .
inline_scan -d ./Dockerfile example-image:latest

Scan image using custom policy bundle, fail script upon failed policy evaluation.

inline_scan -f -p ./policy-bundle.json example-image:latest

Inline image analysis

For use cases where it is desirable to perform image analysis for a locally build container image, and import the image analysis to an existing Anchore Engine installation, we support a methodology using the inline_scan tool. With this technique, you can 'add' an image to your anchore engine service by analyzing any image that is available locally (say, on the docker host on which the image was built). You can then import the analysis data into anchore engine, rather than the regular method where images are pulled from a registry when added to anchore engine.

The only requirements to run the inline_scan script with the 'analyze' operation is the ability to execute Docker commands, network connectivity to an anchore engine API endpoint & bash. We host a versioned copy of this script that can be downloaded directly with curl and executed in a bash pipeline.

• Note - For the rest of this document, USER, PASS, and URL refer to an anchore engine user, password, and engine endpoint URL (http://:/v1) respectively.

To run the script on your workstation, use the following command syntax.

curl -s https://ci-tools.anchore.io/inline_scan-v0.10.0 | bash -s -- analyze -u <USER> -p <PASS> -r <URL> [ OPTIONS ] <FULL_IMAGE_TAG>

Image Identity Selection

In order to perform local analysis and import the image correctly into your existing anchore engine deployment, special attention should be paid to the image identifiers (image id, digest, manifest, full tag name) when performing local analysis. Since image digests are generated from an image 'manifest' which is generated by a container registry, this technique requires that you either specify a digest, ask for one to be randomly 'generated', or supply a valid manifest alongside the image when it is being imported. An image ID can also be supplied if one is available that you would prefer to use. Best practice is to supply these identifiers in whichever way is most appropriate for your use case, resulting in the information being associated with the imported image correctly such that you can refer to it later using these identifiers.

See docs.anchore.com for more usage examples.

ci-tools's Issues

parse_reports uses incorrect json filenames

Based on the the supplied image name in the circleci config, the following files are generated according to the output:

...
Analysis completed!

Successfully generated anchore-reports/api-server_ci-content-os.json.
Successfully generated anchore-reports/api-server_ci-content-npm.json.
Successfully generated anchore-reports/api-server_ci-content-files.json.
Successfully generated anchore-reports/api-server_ci-vuln.json.
Successfully generated anchore-reports/api-server_ci-details.json.
Successfully generated anchore-reports/api-server_ci-policy.json.

However, the parse_reports step references these files:

anchore-reports/image-content-os-report.json
anchore-reports/image-vuln-report.json

Ref: https://github.com/anchore/ci-tools/blob/master/circleci-orbs/anchore-engine/orb.yml#L205-L207

So I get these errors instead:

The following OS packages are installed:
jq: error: Could not open file anchore-reports/image-content-os-report.json: No such file or directory

The following vulnerabilites were found:
jq: error: Could not open file anchore-reports/image-vuln-report.json: No such file or directory

My circleci config:

  local_image_scan:
    executor: anchore/anchore_engine
    steps:
      - checkout
      - run:
          name: Build image for scanning
          command: docker build -t ${CIRCLE_PROJECT_REPONAME}:ci .
      - anchore/analyze_local_image:
          image_name: ${CIRCLE_PROJECT_REPONAME}:ci
          timeout: '500'
          policy_failure: False

vulnerabilities data older than 10 days

Hello,

You state that the vuln database is updated daily but running the inline scan i get the alert the data is older than 10 days. I running the scan agains a debian based image.

Thank you!

inline_scan doesn't seem to honor -b correctly

Followup on #9

The problem

When I try to use the -b flag to point to a policy bundle I get an error that makes me think it doesn't understand the flag. (It tries to scan an image called policy_bundle.json)

Environment

OS X Mojave; bash script calling version 0.6.0 of inline_scan downloaded from github.

Details

When I try to use the -b flag to point to a policy bundle (either by relative path or absolute) I get the following error:

$ anchore-ci-tools/scripts/inline_scan scan -b ../anchore/policy_bundle.json -t 300 -p -r my.docker.registry/my-docker-image@sha256:211de8a383c...
...
time="2019-12-31T20:56:10Z" level=fatal msg="Error loading tar component manifest.json: archive/tar: invalid tar header"
	Ignoring invalid docker archive:  /anchore-engine/policy_bundle.json

Unable to activate policy bundle - /anchore-engine/policy_bundle.json -- using default policy bundle.
...
Error: Policy bundle null not found in DB
HTTP Code: 404
Detail: {'error_codes': []}
...

Actual Behaviour

Seems to run okay but with default policy bundle.

Expected Behaviour

How do you reproduce the error?

Surprisingly it seems somewhat flaky. It never seems to load the policy bundle but it does seem to vary whether it gives the "Error loading tar component" error.

Vulnerability data unavailable for inline scanning

Hi, using the following command for a local image based on UBI8:

curl -s https://ci-tools.anchore.io/inline_scan-v0.3.3 | bash -s --  -t 500 -f <image name>

I receive the following for vulnerabilities:

Gate                   Trigger                               Detail                                                                                     Status
vulnerabilities        stale_feed_data                       The vulnerability feed for this image distro is older than MAXAGE (2) days                 warn          
vulnerabilities        vulnerability_data_unavailable        Feed data unavailable, cannot perform CVE scan for distro: rhel:8.0                        warn

Is there a way to update the data for the inline scanning?

permission denied

The problem

When running in GitLab CI there

• anchore_ci_tools.py -a -r --timeout 500 --image $CI_REGISTRY/hello-world:v1.99.99
  ERROR executing script - Exception: [Errno 13] Permission denied: 'anchore-reports/hello-world_v1.99.99-content-os.json'

Environment

GitLab Ci on gitlab.com

Details

Actual Behaviour

$ anchore_ci_tools.py -a -r --timeout 500 --image $CI_REGISTRY/hello-world:v1.99.99

Image submitted to Anchore Engine: hello-world:v1.99.99
Waiting for analysis to complete...
Status: not_analyzed
Status: analyzing..
Status: analyzed
Analysis completed!
ERROR executing script - Exception: [Errno 13] Permission denied: 'anchore-reports/hello-world_v1.99.99-content-os.json'
Authenticating with credentials from /root/.docker/config.json
00:02
Uploading artifacts...
anchore-reports/*: found 5 matching files
Uploading artifacts to coordinator... ok id=12274 responseStatus=201 Created token=o2iy3TDJ
ERROR: Job failed: exit code 1

Expected Behaviour

How do you reproduce the error?

container_scan_service:
stage: Scan
variables:
ANCHORE_CLI_URL: "http://anchore-engine:8228/v1"
GIT_STRATEGY: none
image: docker.io/anchore/inline-scan:latest
services:

• name: docker.io/anchore/inline-scan:latest
  alias: anchore-engine
  command: ["start"]

script:

• source /opt/rh/rh-python36/enable && rm -rf anchore-reports
• anchore-cli system wait
• anchore-cli image add $CI_REGISTRY/hello-world:v1.99.99
• anchore-cli registry list
• anchore-cli --u admin --p foobar image list
• anchore-cli registry add "$CI_REGISTRY" gitlab-ci-token "$CI_JOB_TOKEN" --skip-validate
• anchore_ci_tools.py -a -r --timeout 500 --image $CI_REGISTRY/hello-world:v1.99.99

artifacts:
when: always
name: ${CI_JOB_NAME}-${CI_COMMIT_REF_NAME}
paths:
- anchore-reports/*

Reports are not copied into workspace when inline_scan is invoked with both -r and -f

The problem
When I run inline_scan with both -r and -f and if the policy evaluation fails, the reports are not copied from container to workspace.
If policy evaluation is successful, the reports are copied from container into the workspace.

Details
It looks like that if -f option is passed, scan exits with 1 and the following code from start_vuln_scan() is not executed:

if [[ "${r_flag}" ]]; then
  echo "Copying scan reports from ${DOCKER_NAME} to ${PWD}/anchore-reports/"
  docker cp "${DOCKER_NAME}:/anchore-engine/anchore-reports/" ./
fi

Actual Behaviour
Reports are not copied from the container and anchore-reports directory in my workspace is empty.

Expected Behaviour
Reports should be copied from container into anchore-reports directory in the workspace.

How do you reproduce the error?

curl -s [...]/inline_scan-v0.6.0 | bash -s -- -f -r -d Dockerfile -b anchore-policy.json IMAGE:TAG"

If the policy evaluation fails, the anchore-reports directory in my workspace is empty.

"no space left on device" occurred.

The problem

I want to perform a Docker imae vulnerability scan without needing to retrieve Docker images from the remote registry, so I ran the Inline vulnerability scanner, but I get the following error "no space left on device".

I checked the capacity with "df -h" and "df -i" and there was enough space.

How do you respond?

The command you executed.
$curl -s https://ci-tools.anchore.io/inline_scan-v0.6.0 | bash -s xxxxxx:v:v:v
xxxxxx:v:v:v : imagename

Details

The following error occurred
failed to register layer:ApplyLayer exit status1 stdout: stderr: write /var/lib/posgresql/data/pg_xlog/000000000100000000000070: no space left on device

anchore_ci_tools.py error /usr/bin/env: python3: No such file or directory

The problem

When running in GitLab CI there

$ anchore_ci_tools.py -a -r --timeout 500 --image $IMAGE_COMMIT
/usr/bin/env: python3: No such file or directory

Environment

GitLab Ci on gitlab.com

Details

Actual Behaviour

Running with gitlab-runner 12.1.0 (de7731dd)
   on test-zip-files usRsAsNe
Using Docker executor with image docker.io/anchore/inline-scan:latest ...
 Starting service docker.io/anchore/inline-scan:latest ...
 Pulling docker image docker.io/anchore/inline-scan:latest ...
 Using docker image sha256:fb97ec7e421ef4b7df63218a3a8650879727c521ecefd121a4ee2fd9f0694ffe for docker.io/anchore/inline-scan:latest ...
 Waiting for services to be up and running...
Running on runner-usRsAsNe-project-14523306-concurrent-0 via 55e872d2a5a8...
00:01
Skipping Git repository setup
00:02
 Skipping Git checkout
 Skipping Git submodules setup
$ anchore-cli system wait
 Starting checks to wait for anchore-engine to be available timeout=-1.0 interval=5.0
 API availability: Checking anchore-engine URL (http://anchore-engine:8228/v1)...
 API availability: Success.
 Service availability: Checking for service set (catalog,apiext,policy_engine,simplequeue,analyzer)...
 Service availability: Checking for service set (catalog,apiext,policy_engine,simplequeue,analyzer)...
 Service availability: Checking for service set (catalog,apiext,policy_engine,simplequeue,analyzer)...
 Service availability: Success.
 Feed sync: Checking sync completion for feed set (vulnerabilities)...
 Feed sync: Success.
 $ anchore-cli registry add "$CI_REGISTRY" gitlab-ci-token "$CI_JOB_TOKEN" --skip-validate
 Registry: registry.gitlab.com
 Name: registry.gitlab.com
 User: gitlab-ci-token
 Type: docker_v2
 Verify TLS: True
 Created: 2020-01-24T19:10:24Z
 Updated: 2020-01-24T19:10:24Z
 $ anchore_ci_tools.py -a -r --timeout 500 --image $IMAGE_COMMIT
 /usr/bin/env: python3: No such file or directory
 ERROR: Job failed: exit code 1

Expected Behaviour

How do you reproduce the error?

container_scan_service:
  stage: post_build
  variables:
    ANCHORE_CLI_URL: "http://anchore-engine:8228/v1"
    GIT_STRATEGY: none
  image: docker.io/anchore/inline-scan:latest
  services:
  - name: docker.io/anchore/inline-scan:latest
    alias: anchore-engine
    command: ["start"]
  
  script:
  - anchore-cli system wait
  - anchore-cli registry add "$CI_REGISTRY" gitlab-ci-token "$CI_JOB_TOKEN" --skip-validate 
  - anchore_ci_tools.py -a -r --timeout 500 --image $IMAGE_COMMIT

  artifacts:
    name: ${CI_JOB_NAME}-${CI_COMMIT_REF_NAME}
    paths:
    - anchore-reports/*

Anchore orb cannot find custom policy_bundle.json file

Summary:
Anchore orb for circleci cannot find custom policy_bundle.json file.

Steps to reproduse:
Add a policy_bundle.json file to [project_name]/.circleci/.anchore/
It's a default folder. The orb should find the file, but you can also add a parameter:
policy_bundle_file_path: /home/circleci/project/.circleci/.anchore/policy_bundle.json

Expected results:
I used a defult policy_bundle.json file that I found here

Actual results
I've got this error:

#!/bin/bash -eo pipefail
(anchore-cli --json policy add "/home/circleci/project/.circleci/.anchore/policy_bundle.json" | jq '.policyId' | xargs anchore-cli policy activate) &> /dev/null || \
  printf "\n%s\n" "Custom policy bundle not found - /home/circleci/project/.circleci/.anchore/policy_bundle.json - using default policy bundle."
if true; then
  anchore-cli evaluate check "anchore-registry:5000/${CIRCLE_PROJECT_REPONAME}:ci" --detail
else
  (set +o pipefail; anchore-cli evaluate check "anchore-registry:5000/${CIRCLE_PROJECT_REPONAME}:ci" --detail | tee /dev/null)
fi

Custom policy bundle not found - /home/circleci/project/.circleci/.anchore/policy_bundle.json - using default policy bundle.

Comments:
My custom policy should pass all vulnerabilities expect critical, but the circleci job feils because the default policy is set to high.

CircleCI pipeline code:

version: 2.1
orbs:
  anchore: anchore/[email protected]
jobs:
  local_image_scan:
    executor: anchore/anchore_engine
    steps:
      - checkout:
          path: ~/project/src/
      - run:
          name: build container
          command: docker build -t ${CIRCLE_PROJECT_REPONAME}:ci ~/project/src/
      - anchore/analyze_local_image:
          image_name: ${CIRCLE_PROJECT_REPONAME}:ci
          timeout: '500'
          policy_failure: True
          policy_bundle_file_path: /home/circleci/project/.circleci/.anchore/policy_bundle.json
      - anchore/parse_reports
workflows:
  scan_image:
    jobs:
      - local_image_scan:
          context: global

Path to policy_bundle.json
/home/user/project/.circleci/.anchore/policy_bundle.json

anchore-inline break in uploading custom policy (-b policy.json)

The problem

Some change has precluded the loading of custom policy (policy.json) for inline scan. only output i receive "Unable to activate policy bundle - /anchore-engine/policy.json -- using default policy bundle.

Environment

anchore/inline-scan:latest (9da71f6b0958)

Details

Actual Behaviour

Expected Behaviour

How do you reproduce the error?

Error using anchore-engine v0.3.2

Image analysis fails on v0.3.2 of anchore-engine, when called with the anchore_ci_tools.py script, causing the following skopeo error:
[service:worker] 2019-01-18 03:12:30+0000 [-] anchore_engine.clients.skopeo_wrapper.SkopeoError: Error encountered in skopeo operation. cmd=/bin/sh -c skopeo copy --src-tls-verify=false docker://docker.io/alpine@sha256:3d2e482b82608d153a374df3357c0291589a61cc194ec4a9ca2381073a17f58e oci:/tmp/a35cc446-553a-4aad-b0c9-ef9b058a5b55/raw:image, rc=1, stdout=b'Getting image source signatures\n', stderr=b'time="2019-01-18T03:12:30Z" level=fatal msg="error creating progress-bar pool: Can\'t set terminal settings: input/output error" \n'

SARIF report output

The problem

The CI tools do not output a SARIF formatted report.

Environment

Running GitLab CI.

Details

I see the Anchore GitHub scan action has a SARIF formatted report output. Are there plans to add this to this project, so other CIs can get the same report?

Actual Behaviour

No SARIF report option.

Expected Behaviour

SARIF report option.

How do you reproduce the error?

clean up inline_scan output

inline_scan script has some output that does not need to be shown all the time. Update the script to only show important output unless the verbose option is specified. Also, make some of the error messages more verbose for easier debugging.

Multiple containers won't scan

curl -s https://raw.githubusercontent.com/anchore/ci-tools/master/scripts/inline_scan | bash -s -- -t 35000 -p -r alpine:latest ubuntu:latest centos:latest

gives the following output

Pulling image -- alpine:latest ubuntu:latest centos:latest
"docker pull" requires exactly 1 argument.
See 'docker pull --help'.

Usage:  docker pull [OPTIONS] NAME[:TAG|@DIGEST]

Pull an image or a repository from a registry

WARNING - Please pull remote image, or build/tag all local images before attempting analysis again


        ERROR - no local docker images specified in script input: bash alpine:latest ubuntu:latest centos:latest


Anchore Engine Inline Scanner/Analyzer --

  Wrapper script for performing vulnerability scan or image analysis on local docker images, utilizing the Anchore Engine inline_scan container.
  For more detailed usage instructions use the -h option after specifying scan or analyze.

    Usage: bash <scan|analyze> [ OPTIONS ]

This has broken a couple of our CI scans. I'm assuming I'm using the script correctly?

Failed to run a concurrent scanning jobs for the same inline scanner

The problem

When a second scanning job is starting while a first job is still in progress, the second job gets failed scan after a timeout.

Environment

Gitlab+K8S

Details

I am using one scanner for several microservice pipelines, I am getting sometimes "scan job failed". I assume that this is happening when there is already one job in progress and the second one is waiting for the first one to complete its job till it gets timeout...

Actual Behaviour

The second scan job gets failed status as a result...

Expected Behaviour

Expecting to run multiple scanning jobs at the same time. How can I setup my inline scanner for that?

How do you reproduce the error?

Run several pipelines for different microservices one by one...

possible to expose detailed policy evaluation?

The problem

Not a problem but a feature request. it would be very helpful to be able to get detailed json output from policy evaluation from anchore-inline as well as anchore-engine

Environment

anchore/inline-scan latest 966822e88cf6 15 hours ago

Details

is there a way to expose details of a analysis with anchore-inline like the below command from anchore-engine?

$ anchore-cli --debug --json evaluate check <target_image> --detail

inline-scan:latest Broken Image

First time using this image but found https://anchore.freshdesk.com/support/solutions/articles/36000058199-gitlab-integration and using the docker.io/anchore/inline-scan:latest image is broken. I was able to run it by using an older tag docker.io/anchore/inline-scan:v0.3.4.

Looks like the /docker-entrypoint.sh doesn't exist:

$ anchore-cli system wait
/usr/local/bin/anchore-cli: line 3: /docker-entrypoint.sh: No such file or directory

Running with gitlab-runner 12.0.2 (d0b76032)
  on docker-auto-scale ed2dce3a
Using Docker executor with image docker.io/anchore/inline-scan:latest ...
Starting service docker.io/anchore/inline-scan:latest ...
Pulling docker image docker.io/anchore/inline-scan:latest ...
Using docker image sha256:fc0e5d89a4e419e2dd6511122aec6741f661f4f635d0d3b96ae81c1c49970946 for docker.io/anchore/inline-scan:latest ...
Waiting for services to be up and running...

*** WARNING: Service runner-ed2dce3a-project-13065579-concurrent-0-docker.io__anchore__inline-scan-0 probably didn't start properly.

Health check error:
service "runner-ed2dce3a-project-13065579-concurrent-0-docker.io__anchore__inline-scan-0-wait-for-service" timeout

Health check container logs:


Service container logs:
2019-07-08T18:01:26.216621216Z Starting Postgresql... Postgresql started successfully!
2019-07-08T18:01:29.354938684Z Starting Docker registry... Docker registry started successfully!
2019-07-08T18:01:29.355065194Z Starting Anchore Engine...

*********

Running on runner-ed2dce3a-project-13065579-concurrent-0 via runner-ed2dce3a-srm-1562608667-60e73e90...
Skipping Git repository setup
Skipping Git checkout
Skipping Git submodules setup
Downloading artifacts for container_build (247493437)...
Downloading artifacts from coordinator... ok        id=247493437 responseStatus=200 OK token=Q3zDfJnK
$ anchore-cli system wait
/usr/local/bin/anchore-cli: line 3: /docker-entrypoint.sh: No such file or directory
ERROR: Job failed: exit code 1

Running with gitlab-runner 12.0.2 (d0b76032)
  on docker-auto-scale 0277ea0f
Using Docker executor with image docker.io/anchore/inline-scan:v0.3.4 ...
Starting service docker.io/anchore/inline-scan:v0.3.4 ...
Pulling docker image docker.io/anchore/inline-scan:v0.3.4 ...
Using docker image sha256:63ee3fbfe207a726f870efcc124d70c0466542724bc2853abd8e7133b4609b8e for docker.io/anchore/inline-scan:v0.3.4 ...
Waiting for services to be up and running...
Running on runner-0277ea0f-project-13065579-concurrent-0 via runner-0277ea0f-srm-1562609402-97255c0d...
Skipping Git repository setup
Skipping Git checkout
Skipping Git submodules setup
Downloading artifacts for container_build (247505785)...
Downloading artifacts from coordinator... ok        id=247505785 responseStatus=200 OK token=FpCmNmkZ
$ anchore-cli system wait
Starting checks to wait for anchore-engine to be available timeout=-1.0 interval=5.0
API availability: Checking anchore-engine URL (http://anchore-engine:8228/v1)...
API availability: Success.
Service availability: Checking for service set (catalog,apiext,policy_engine,simplequeue,analyzer)...
Service availability: Checking for service set (catalog,apiext,policy_engine,simplequeue,analyzer)...
Service availability: Checking for service set (catalog,apiext,policy_engine,simplequeue,analyzer)...
Service availability: Success.
Feed sync: Checking sync completion for feed set (vulnerabilities)...
Feed sync: Success.
$ anchore-cli registry add "$CI_REGISTRY" gitlab-ci-token "$CI_JOB_TOKEN" --skip-validate
Registry: registry.gitlab.com
User: gitlab-ci-token
Type: docker_v2
Verify TLS: True
Created: 2019-07-08T18:14:13Z
Updated: 2019-07-08T18:14:13Z


$ anchore_ci_tools.py -a -r --timeout 500 --image $IMAGE_TAG

Image submitted to Anchore Engine: mysql:devops-pipeline-build-image
Waiting for analysis to complete...

	Status: not_analyzed
	Status: analyzing............
	Status: analyzed

Analysis completed!

Successfully generated anchore-reports/mysql_devops-pipeline-build-image-content-os.json.
Successfully generated anchore-reports/mysql_devops-pipeline-build-image-content-files.json.
Successfully generated anchore-reports/mysql_devops-pipeline-build-image-vuln.json.
Successfully generated anchore-reports/mysql_devops-pipeline-build-image-details.json.
Successfully generated anchore-reports/mysql_devops-pipeline-build-image-policy.json.
Uploading artifacts...
anchore-reports/*: found 5 matching files          
Uploading artifacts to coordinator... ok            id=247505786 responseStatus=201 Created token=LYyXejzs
Job succeeded

More options for report generation (non-os mode)

Same as the anchore-cli with the non-os option, it will be amazing if the same option was available for inline scan.

I hope it will be possible.

Regards !

CircleCI orb is not loading policy bundle

The problem

When I specify a path to a policy bundle using the policy_bundle_file_path param, I get "Failed to activate policy bundle" even though the policy bundle file is there.

Environment

CircleCI, anchore/[email protected] CircleCI Orb.

Details

I'm going to guess it's because of the missing Docker image name in this line:

ci-tools/circleci-orbs/anchore-engine/src/commands/policy_evaluation.yml

Line 30 in e595ab4

(docker exec -e BUNDLE_PATH="$BUNDLE_PATH" -i anchore-engine bash -lc 'anchore-cli --json policy add /anchore-engine/$(basename "$BUNDLE_PATH")' | jq '.policyId' | xargs docker exec -i bash -lc 'anchore-engine anchore-cli policy activate') || \

The surrounding output also implies that:

fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/2) Installing oniguruma (6.9.4-r0)
�7  0%                                             �8(2/2) Installing jq (1.6-r0)
�7 49% #####################                       �8�7 73% ################################            �8�7100% ############################################�8Executing busybox-1.31.1-r8.trigger
OK: 34 MiB in 28 packages
Error: No such container: bash

Failed to activate policy bundle /root/project/.anchore/anchore_default_bundle.json - using default policy.
^@Image Digest: sha256:47...
Full Tag: ****************************************************************:commit-61...
Image ID: 24...
Status: fail

Actual Behaviour

Anchore engine does not load the specified policy

Expected Behaviour

Anchore engine loads the specified policy

How do you reproduce the error?

Specify a policy

False Negative vulnerability

The problem

With ubunut:16.04 looks like we are missing alert for CVE-2020-13776 on systemd that is vulnerable according to canonical.

Environment

anchore/inline-scan latest c97d1d2ed9ec 7 hours ago 5.9GB

target image:
ubuntu 16.04 005d2078bdfa 6 weeks ago 125MB

Details

This ubuntu 16.04 standard image has (systemd 229-4ubuntu21.27) package which Canonical has an advisory with a "Medium" severity rating. https://people.canonical.com/~ubuntu-security/cve/2020/CVE-2020-13776.html.

though Canonical calls out needs to be triaged, this should still throw an alert.

Actual Behaviour

While there are number of alerts, we do not see one for CVE-2020-13776 for this version of systemd.

Expected Behaviour

We should see an alert for CVE-2020-13776 for systemd on this image; unless there is some logic suppressing unremediable alerts or very new alerts?

How do you reproduce the error?

1. $ docker pull ubuntu:16.04
2. scan with inline-scan and view how CVE-2020-13776 is not in the list
3. docker run -it ubunut:16.04 /bin/bash
4. dpkg -l systemd
   this confirms the version.

thanks!
nik

vuln feeds haven't been updated for anchore-inline image since at least nov 29

The problem

The vulnerability feeds (nvd, canonical, etc) appear to be stale.

Environment

Details

Actual Behaviour

Expected Behaviour

How do you reproduce the error?

Get CircleCI Orb certified

Hello,
I don't know about the process of certifying circleci orbs, but is there a chance to get the anchore-engine orb certified?

[circle-ci][orb] "Install anchore tools" step fails

Seeing this issue during local image check that blocks the CI:

#!/bin/bash -eo pipefail
curl -o /tmp/anchore_ci_tools.py https://raw.githubusercontent.com/anchore/ci-tools/v0.3.2/scripts/anchore_ci_tools.py
chmod +x /tmp/anchore_ci_tools.py
pyenv global 3.6.5
pip install --upgrade pip
pip install --upgrade docker-compose
pip install --upgrade anchorecli
sed -i 's|/usr/bin/python3|/opt/circleci/.pyenv/shims/python3|' /tmp/anchore_ci_tools.py
ln -s /tmp/anchore_ci_tools.py /home/circleci/bin/anchore_ci_tools
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

100 13404  100 13404    0     0  57973      0 --:--:-- --:--:-- --:--:-- 58025
Requirement already up-to-date: pip in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (19.0.3)
Requirement already up-to-date: docker-compose in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (1.24.0)
Requirement already satisfied, skipping upgrade: cached-property<2,>=1.2.0 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from docker-compose) (1.5.1)
Requirement already satisfied, skipping upgrade: websocket-client<1.0,>=0.32.0 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from docker-compose) (0.56.0)
Requirement already satisfied, skipping upgrade: PyYAML<4.3,>=3.10 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from docker-compose) (4.2b1)
Requirement already satisfied, skipping upgrade: six<2,>=1.3.0 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from docker-compose) (1.12.0)
Requirement already satisfied, skipping upgrade: docopt<0.7,>=0.6.1 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from docker-compose) (0.6.2)
Requirement already satisfied, skipping upgrade: requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from docker-compose) (2.20.0)
Requirement already satisfied, skipping upgrade: texttable<0.10,>=0.9.0 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from docker-compose) (0.9.1)
Requirement already satisfied, skipping upgrade: dockerpty<0.5,>=0.4.1 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from docker-compose) (0.4.1)
Requirement already satisfied, skipping upgrade: jsonschema<3,>=2.5.1 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from docker-compose) (2.6.0)
Requirement already satisfied, skipping upgrade: docker[ssh]<4.0,>=3.7.0 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from docker-compose) (3.7.2)
Requirement already satisfied, skipping upgrade: idna<2.8,>=2.5 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1->docker-compose) (2.7)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1->docker-compose) (2019.3.9)
Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1->docker-compose) (1.23)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1->docker-compose) (3.0.4)
Requirement already satisfied, skipping upgrade: docker-pycreds>=0.4.0 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from docker[ssh]<4.0,>=3.7.0->docker-compose) (0.4.0)
Requirement already satisfied, skipping upgrade: paramiko>=2.4.2; extra == "ssh" in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from docker[ssh]<4.0,>=3.7.0->docker-compose) (2.4.2)
Requirement already satisfied, skipping upgrade: pynacl>=1.0.1 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from paramiko>=2.4.2; extra == "ssh"->docker[ssh]<4.0,>=3.7.0->docker-compose) (1.3.0)
Requirement already satisfied, skipping upgrade: cryptography>=1.5 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from paramiko>=2.4.2; extra == "ssh"->docker[ssh]<4.0,>=3.7.0->docker-compose) (2.6.1)
Requirement already satisfied, skipping upgrade: bcrypt>=3.1.3 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from paramiko>=2.4.2; extra == "ssh"->docker[ssh]<4.0,>=3.7.0->docker-compose) (3.1.6)
Requirement already satisfied, skipping upgrade: pyasn1>=0.1.7 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from paramiko>=2.4.2; extra == "ssh"->docker[ssh]<4.0,>=3.7.0->docker-compose) (0.4.5)
Requirement already satisfied, skipping upgrade: cffi>=1.4.1 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from pynacl>=1.0.1->paramiko>=2.4.2; extra == "ssh"->docker[ssh]<4.0,>=3.7.0->docker-compose) (1.12.3)
Requirement already satisfied, skipping upgrade: asn1crypto>=0.21.0 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from cryptography>=1.5->paramiko>=2.4.2; extra == "ssh"->docker[ssh]<4.0,>=3.7.0->docker-compose) (0.24.0)
Requirement already satisfied, skipping upgrade: pycparser in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from cffi>=1.4.1->pynacl>=1.0.1->paramiko>=2.4.2; extra == "ssh"->docker[ssh]<4.0,>=3.7.0->docker-compose) (2.19)
Requirement already up-to-date: anchorecli in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (0.3.4)
Requirement already satisfied, skipping upgrade: click==7.0 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from anchorecli) (7.0)
Requirement already satisfied, skipping upgrade: prettytable==0.7.2 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from anchorecli) (0.7.2)
Requirement already satisfied, skipping upgrade: requests==2.20.0 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from anchorecli) (2.20.0)
Requirement already satisfied, skipping upgrade: urllib3==1.23 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from anchorecli) (1.23)
Requirement already satisfied, skipping upgrade: python-dateutil==2.7.3 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from anchorecli) (2.7.3)
Requirement already satisfied, skipping upgrade: PyYAML==4.2b1 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from anchorecli) (4.2b1)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from requests==2.20.0->anchorecli) (2019.3.9)
Requirement already satisfied, skipping upgrade: idna<2.8,>=2.5 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from requests==2.20.0->anchorecli) (2.7)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from requests==2.20.0->anchorecli) (3.0.4)
Requirement already satisfied, skipping upgrade: six>=1.5 in /opt/circleci/.pyenv/versions/3.6.5/lib/python3.6/site-packages (from python-dateutil==2.7.3->anchorecli) (1.12.0)
ln: failed to create symbolic link '/home/circleci/bin/anchore_ci_tools': File exists
Exited with code 1

build config:

  "python371_security_check":
    executor: anchore/anchore_engine
    working_directory: ~/fdk-python
    steps:
      - checkout
      - run:
          name: Python 3.7.1 build
          command: |
            ./build-images.sh 3.7.1
            mkdir -p /home/circleci/project/.circleci/.anchore/
            cp ~/fdk-python/.circleci/.anchore/policy_bundle.json /home/circleci/project/.circleci/.anchore/policy_bundle.json
      - anchore/analyze_local_image:
          image_name: fnproject/python:3.7.1-dev
          timeout: '500'
          policy_failure: true
      - anchore/analyze_local_image:
          image_name: fnproject/python:3.7.1
          timeout: '500'
          policy_failure: true
      - anchore/parse_reports

see build log for more details https://circleci.com/gh/fnproject/fdk-python/327

custom analyzer_config.yaml support?

The problem

With the anchore-engine, users can customize the analyzer_config.yaml file to do additional regex search patterns. Will anchore-inline support this feature as well?

Environment

Details

Actual Behaviour

Expected Behaviour

How do you reproduce the error?

Add example using inline_scan in BitBucket pipeline

Atlassian has reported that inline_scan works in BitBucket pipeline. Create an example config showing an image build/scan pipeline.

Add Issue and Pull Request template

Description

I believe the project could include Issue and Pull Request templates, making it easier to open one of them.

Cannot Use json key in $docker_pass in circleci orb

I'm trying to login using json key, but getting error:

anchore-cli registry add eu.gcr.io $DOCKER_USER $DOCKER_PASS --skip-validate
Usage: anchore-cli registry add 
           [OPTIONS] REGISTRY REGISTRY_USER
           REGISTRY_PASS
Try "anchore-cli registry add --help" for help.

Error: Got unexpected extra arguments ("type": "service_account", "project_id": "my_project", "private_key_id": "{my_json_key}")
Exited with code 2

Circleci pipeline:

workflows:
  scan_image:
    jobs:
      - anchore/image_scan:
          image_name: eu.gcr.io/my_project/my_image:tag
          private_registry: true
          registry_name: eu.gcr.io
          registry_user: _json_key
          registry_pass: $_gcr_json_key
...

How can I login with json key to a private registry?

[feature] allow custom path for policy bundle

Follow-up on anchore/anchore-engine#174

intro

CircleCI Anchore orb allows using a bundle file that stored at .circleci/.anchore/policy_bundle.json.

However, there's a problem because anchore/analyze_local_image attempts to load custom bundle file from the following path:

/home/circleci/project/.circleci/.anchore/policy_bundle.json

when at most of the times it's not there and here's very simple circleci job template that breaks anchore orb:

  "python36_security_check":
    executor: anchore/anchore_engine
    working_directory: ~/go/src/github.com/fnproject/fdk-python
    steps:
      - checkout
      - run:
          name: Python 3.6 build
          command: |
            ./build-images.sh 3.6
      - anchore/analyze_local_image:
          image_name: fnproject/python:3.6-dev
          timeout: '500'
          policy_failure: true
      - anchore/analyze_local_image:
          image_name: fnproject/python:3.6
          timeout: '500'
          policy_failure: true
      - anchore/parse_reports

as you may notice, there's working_directory: ~/go/src/github.com/fnproject/fdk-python that changes working directory and make impossible to find bundle file for anchore:

#!/bin/bash -eo pipefail
(anchore-cli --json policy add /home/circleci/project/.circleci/.anchore/policy_bundle.json | jq '.policyId' | xargs anchore-cli policy activate) &> /dev/null || \
  printf "\n%s\n" "Custom policy bundle not found - /home/circleci/project/.circleci/.anchore/policy_bundle.json - using default policy bundle."
if true; then
  anchore-cli evaluate check anchore-registry:5000/fnproject/python:3.6-dev --detail
else
  (set +o pipefail; anchore-cli evaluate check anchore-registry:5000/fnproject/python:3.6-dev --detail | tee /dev/null)
fi

Custom policy bundle not found - /home/circleci/project/.circleci/.anchore/policy_bundle.json - using default policy bundle.

API parameter should be policyId

The Jenkins direct api example uses bundle_id, the correct parameter seems to be policyId.

ci-tools/examples/jenkins/pipeline/direct_api/Jenkinsfile

Line 113 in a6ea370

cmd = "${curler} -o ${output} -X GET '${ANCHORE_ENGINE_URL}/images/${image_digest}/check?bundle_id=${ANCHORE_ENGINE_POLICY_BUNDLE_ID}&tag=${TAG_TO_BE_ANALYZED}&detail=true'"

Update README and inline_scan script to include deprecation notice

Now that Syft and Grype are available we will be deprecating the inline_scan script since Grype provides a better in-pipeline scanning experience.

This issue is to get the text for that notice finalized and included in output from the script itself so that users are aware of the timeline and how to transition effectively to the new methods.

Analyze image based on scratch

Hello everyone!
I hope you are doing well.

The problem

I've tried to analyze image is based on scratch and this proccess had been failed

Environment

Linux 5.4.0-45-generic #49~18.04.2-Ubuntu
docker-compose version 1.17.1, build unknown
Docker version 19.03.6, build 369ce74a3c

Details

1. curl https://docs.anchore.com/current/docs/engine/quickstart/docker-compose.yaml > docker-compose.yaml
2. docker-compose up -d
3. download inline_scan.sh (version 0.8.0)
   4.Create dockerfile

FROM wrouesnel/postgres_exporter:v0.8.0

5. build images is based on scratch (I've used wrouesnel/postgres_exporter:v0.8.0)

docker build -t  test-exporter:v0.8.0 . -f postgres-exporter.dockerfile

6. Try analyze

./inline_scan.sh analyze -r localhost:8228 -u admin -p foobar -V -g -f postgres-exporter.dockerfile test-exporter:v0.8.0

Actual Behaviour

I've got an error

[MainThread] [anchore_engine.common.helpers/extract_analyzer_content()] [ERROR] could not extract/parse content info - exception: 'package_list'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/anchore_manager/cli/analyzers.py", line 188, in exec
    image_content_data[content_type] = anchore_engine.common.helpers.extract_analyzer_content(image_data, content_type, manifest=input_manifest_data)
  File "/usr/local/lib/python3.6/site-packages/anchore_engine/common/helpers.py", line 331, in extract_analyzer_content
    raise err
  File "/usr/local/lib/python3.6/site-packages/anchore_engine/common/helpers.py", line 292, in extract_analyzer_content
    return extract_os_content(idata)
  File "/usr/local/lib/python3.6/site-packages/anchore_engine/common/helpers.py", line 207, in extract_os_content
    if 'pkgs.allinfo' in image_data['imagedata']['analysis_report']['package_list']:
KeyError: 'package_list'
...
...
...
[MainThread] [anchore_manager.cli.analyzers/exec()] [ERROR] Unable to determine content_type, will fallback to {}
[MainThread] [anchore_engine.common.helpers/extract_analyzer_content()] [ERROR] could not extract/parse content info - exception: 'package_list'
[MainThread] [anchore_manager.util.logging/log_error()] [ERROR] Error: 'package_list'

Expected Behaviour

• printf '%s\n' ' Analysis complete!'
  Analysis complete!

How do you reproduce the error?

It works fine when I use anchore-cli in official anchore docker image

anchore-cli image add wrouesnel/postgres_exporter:v0.8.0
>Analysis Status: not_analyzed

anchore-cli image wait wrouesnel/postgres_exporter:v0.8.0
>Analysis Status: analyzed

anchore-cli image vuln wrouesnel/postgres_exporter:v0.8.0 all
>  __nothing__