microsoft / codeql-container

Prepackaged and precompiled github codeql container for rapid analysis, deployment and development.

CodeQL Container

Note: CodeQL container is currently in public preview. Please report any bugs to https://github.com/microsoft/codeql-container/issues. Current version of CodeQL only works for interpreted languages. We will add compiled languages support in future versions.

The CodeQL Container is a project aimed at making it easier to start using CodeQL (more about codeQL at https://github.com/github/codeql). This project contains a Docker file which builds a container with the latest version of codeql-cli, and the latest codeql queries precompiled. It also contains automation to keep the toolchain in the container updated. You can use this container to:

• Start using codeql-cli and run queries on your projects without installing it on your local machine.
• Use it as an environment to develop codeql queries and test them.
• Test how the queries perform in windows and linux environments (and more...)

We shall continue to add more features and would be happy to accept contributions from the community.

TL;DR

Analyze the python project django located in the folder /tmp/django by running the security and quality QL pack on it:

/scripts/unix/analyze_security.sh /tmp/django/src /tmp/django/results python

The results will be stored in /tmp/django/results/issues.sarif.

Analyze the Javascript project express located in /tmp/express/src by running the extended security QL pack on it:

scripts/unix/run_qlpack.sh /tmp/express/src /tmp/express/results javascript security-extended

The results will be stored in /tmp/express/results/issues.sarif

To find a list of QL packs installed in the container:

docker run --rm --name codeql-container -e CODEQL_CLI_ARGS="resolve qlpacks"  mcr.microsoft.com/cstsectools/codeql-container

Downloading a pre-built container

We keep updating the docker image periodically and uploading it to the Microsoft Container Registry at: mcr.microsoft.com/cstsectools/codeql-container.

You can pull the image by running the command:

$ docker pull mcr.microsoft.com/cstsectools/codeql-container

Building the container from Dockerfile

Building the container should be pretty straightforward.

git clone https://github.com/microsoft/codeql-container
cd codeql-container
docker build . -f Dockerfile -t codeql-container

Basic Usage

The codeQL container executes one codeQL command per invocation. We designed it this way because it makes it easy for the user to run any codeQL command, and not be bound by the automation scripts inside the container.

The basic example format of the container invocation is as follows:

$ docker run --rm --name codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS=<query run...> mcr.microsoft.com/cstsectools/codeql-container

where /dir/to/analyze contains the source files that have to be analyzed, and /dir/for/results is where the result output needs to be stored, and you can specify CODEQL_CLI_ARGS environment variable for specific QL packs to be run on the provided code, among other things. The CODEQL_CLI_ARGS will be passed over to codeQL command line as it is.

For more information on CodeQL and QL packs, please visit https://www.github.com/github/codeql.

CODEQL_CLI_ARGS are the arguments that will be directly passed on to the codeql-cli. For example, in this case, if we supply:

CODEQL_CLI_ARGS="database create /opt/results/source_db -s /opt/src"

it will create a codeQL db of your project (in /dir/to/analyze ) in the /dir/for/results folder.

Note: If you map your source volume to some other mount point other than /opt/src, you will have to make the corresponding changes in the CODEQL_CLI_ARGS.

There are some additional docker environment flags that you can set/unset to control the execution of the container:

• CHECK_LATEST_CODEQL_CLI - If there is a newer version of codeql-cli, download and install it
• CHECK_LATEST_QUERIES - if there is are updates to the codeql queries repo, download and use it
• PRECOMPILE_QUERIES - If we downloaded new queries, precompile all new query packs (query execution will be faster)

WARNING: Precompiling query packs might take a few hours, depending on speed of your machine and the CPU/memory limits (if any) you have placed on the container.

Since CodeQL first creates a database of the code representation, and then analyzes the said database for issues, we need to invoke the container more than once to analyze a source code repo. (Since the container only executes one codeQL command per invocation.)

For example, if you want to analyze a python project source code placed in /dir/to/analyze (or C:\dir\to\analyze for example, in Windows), to analyze and get a SARIF result file, you will have to run:

# create the codeql db
$ export language="python"
$ docker run --rm --name codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS="database create --language=${language} /opt/results/source_db -s /opt/src" mcr.microsoft.com/cstsectools/codeql-container

# upgrade the db if necessary
$ docker run --rm --name codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS=" database upgrade /opt/results/source_db" mcr.microsoft.com/cstsectools/codeql-container

# run the queries in the qlpack
$ docker run --rm --name codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS="database analyze --format=sarifv2 --output=/opt/results/issues.sarif /opt/results/source_db ${language}-security-and-quality.qls" mcr.microsoft.com/cstsectools/codeql-container

For more information on CodeQL and QL packs, please visit https://www.github.com/github/codeql.

Convenience Scripts

Analyzing a source directory takes multiple invocations of the container, as mentioned above. To help with that, we've built some scripts for convenience, which does these invocations for you. These scripts are in the scripts folder, under their respective platforms (unix or windows).

analyze_security.sh

scripts/unix/analyze_security.sh (or scripts/windows/analyze_security.bat for windows) runs the Security and Quality QL pack suite on your project. This is how you would run it:

scripts/unix/analyze_security.sh /path/to/analyze /path/to/results language

For example for the python project can be analyzed thus:

/scripts/unix/analyze_security.sh /tmp/django/src /tmp/django/output python

for JavaScript:

/scripts/unix/analyze_security.sh /tmp/express/src /tmp/express/output javascript

run_qlpack.sh

If you know which QL suite you would like to run on the code, use scripts/unix/run_qlpack.sh (or scripts/windows/run_qlpack.bat for windows).

scripts/unix/run_qlpack.sh /path/to/analyze /path/to/results language qlpack

For example, on windows:

scripts\windows\run_ql_suite.bat e:\temp\express\src e:\temp\express\results javascript code-scanning 

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

codeql-container's Issues

Docker built: Windows line endings break bash script

When I try to build the container on Windows, everything seems to work, but when I run it, I get:

PS D:\temp\codeql-container> docker run -it -t codeql:latest /bin/bash
/usr/bin/env: ‘python3\r’: No such file or directory
Error 127,b'',None from command.

That middle line is because the files have Windows line endings instead of Linux ones. You can fix this by installing dos2unix and then running it in the container as its being built.

codeql-container MCR / Tag Listing

This codeql-container is a fantastic tool for making the execution of codeql and necessary dependencies simple. However I'd like to pin to a particular version to prevent breaking changes (i.e. such as the introduction of a non-root user). I'm struggling to find the associated tagging strategy... I see some images with dates and iterations: 2022-05-29_07.15

I'm not particular about what the strategy should be, just that I can easily reference and identify tags for specific images when needed. That is difficult today unless you pull down all images (which is a lot and are massive). Tags are not available for reference on Dockerhub... and I was hoping I'd find this in the MCR catalog - but don't see it there either.

Is it possible to get this exposed on MCR to be able to view tags?
https://mcr.microsoft.com/en-us/catalog

Can the tags be associated to releases when publishing a new version to be visible on GitHub (if not available to publish in MCR catalog)?

Require signed commits on `main`

Protect matching branches

• (Select) Require signed commits
  Commits pushed to matching branches must have verified signatures.

• (Select) Include administrators
  Enforce all configured restrictions above for administrators.

Refer to "About protected branches" for more information.

CodeQL runs slower if the container runs as a user other than root

I've extended the Dockerfile to make some changes to the startup.py file required for my use case. I changed the Dockerfile to run as another user instead of root, but I noticed that the code scanning takes significantly longer - hours, instead of minutes - to complete.

Here is the Dockerfile I'm using:

FROM mcr.microsoft.com/cstsectools/codeql-container

ARG USERNAME=codeql

RUN adduser --system $USERNAME
RUN apt update && apt install -y git

EXPOSE 5000

WORKDIR /usr/local/startup_scripts/

COPY requirements.txt ./
RUN pip install -r ./requirements.txt

COPY CodeQL/main.py ./startup_server.py

USER ${USERNAME}

ENTRYPOINT ["python3", "-u", "/usr/local/startup_scripts/startup_server.py"]

Any help would be really appreciated! 😄

Add "nodejs" to apt install line in Dockerfile to support Typescript scanning

...otherwise, this error is generated

Initializing database at /opt/results/source_db.
Running build command: []
A fatal error occurred: Exit status 1 from command: [/usr/local/codeql-home/codeql/javascript/tools/autobuild.sh]
Error 2 executing from command.
Exiting...
Command Output: 
[2022-09-12 20:45:25] [build-stdout] Single-threaded extraction.
[2022-09-12 20:45:25] [build-stderr] Could not start Node.js. It is required for TypeScript extraction.
[2022-09-12 20:45:25] [build-stderr] Please install Node.js and ensure 'node' is on the PATH.
[2022-09-12 20:45:25] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/usr/local/codeql-home/codeql/javascript/tools/autobuild.sh])

From https://trganda.github.io/posts/codeql/part-ii/#javascript-and-typescript

Creating databases for JavaScript requires no additional dependencies, but if the project includes TypeScript files, you must install Node.js 6.x or later.

Solution: Modify Dockerfile and add nodejs to the apt-get install line.

codeql-container/Dockerfile

Line 14 in 4632664

apt-get install -y --no-install-recommends \

Error 126: execute permission of setup.py

Following the official instructions, I met the following errors when executing the docker run ... command in Basic Usage of the readme, whenever I build the image myself or just pull the image from the Microsoft Container Registry.

/bin/sh: 1: /usr/local/startup_scripts/setup.py: Permission denied Error 126 executing from command. Exiting...

codeql-container/container/startup.py

Lines 36 to 38 in 073695d

	run_result = check_output_wrapper(
	f"{scripts_dir}/setup.py {setup_script_args}",
	shell=True).decode("utf-8")

I reviewed the code of startup.py and Dockerfile. I think this is due to the lack of execute permission of setup.py. Then I chmod the permission, and the problem is solved.

Use `--build-arg` to specify versions during the build process

• codeql-container/Dockerfile

  Line 58 in d0e531a

  RUN python3 /usr/local/startup_scripts/get-latest-codeql-version.py > /tmp/codeql_version

• codeql-container/Dockerfile

  Lines 65 to 66 in d0e531a

  	RUN git clone https://github.com/github/codeql ${CODEQL_HOME}/codeql-repo && \
  	git --git-dir ${CODEQL_HOME}/codeql-repo/.git log --pretty=reference -1 > /opt/codeql/codeql-repo-last-commit

• codeql-container/Dockerfile

  Lines 67 to 68 in d0e531a

  	RUN git clone https://github.com/github/codeql-go ${CODEQL_HOME}/codeql-go-repo && \
  	git --git-dir ${CODEQL_HOME}/codeql-go-repo/.git log --pretty=reference -1 > /opt/codeql/codeql-go-repo-last-commit

• codeql-container/Dockerfile

  Lines 70 to 73 in d0e531a

  	RUN CODEQL_VERSION=$(cat /tmp/codeql_version) && \
  	wget -q https://github.com/github/codeql-cli-binaries/releases/download/${CODEQL_VERSION}/codeql-linux64.zip -O /tmp/codeql_linux.zip && \
  	unzip /tmp/codeql_linux.zip -d ${CODEQL_HOME} && \
  	rm /tmp/codeql_linux.zip

These occurrences can be updated to vastly improve the idempotence of building the Docker image. Right now, running docker build produces different images based on when the command is run. It could change quite a bit over the course of a week.

Docker build fails: Could not resolve library path for /usr/local/codeql-home/codeql-repo/ruby/ql/src

% git describe --always
beb1e2c

% docker build . -f Dockerfile -t codeql-container
[...]
[compiling queries]
[...]
Done [886/911 comp 3.4s] /usr/local/codeql-home/codeql-repo/python/ql/src/Metrics/FLinesOfComments.ql.
ERROR: Referenced pack 'codeql/suite-helpers' not found. (/usr/local/codeql-home/codeql-repo/ruby/ql/src/qlpack.yml:1,1-1)
A fatal error occurred: Could not resolve library path for /usr/local/codeql-home/codeql-repo/ruby/ql/src
The command '/bin/sh -c codeql query compile --threads=0 ${CODEQL_HOME}/codeql-repo//ql/src/codeql-suites/.qls' returned a non-zero code: 2

Also: the image on docker hub is 1 year old. Would it be possible to publish a newer one?

Thanks!

Does this work with GHES?

We are using GitHub Enterprise. Is there a way to make this work with GHES? At first glance of the code it doesn't appear there is a way to over-ride this going to gh.com.

codeQL sarif Upload

hello guys, this is not an issue just a question..
how can we uoload .sarif file using this container is there any option provided?
because in codeql cli there is an upload command available..
or do we have to use the curl command?

Thanks :)

NameError: name 'ERROR_EXECUTING_CODEQL' is not defined

❯ docker run --rm --name codeql-container -e CODEQL_CLI_ARGS="resolve qlpacks"  mcr.microsoft.com/cstsectools/codeql-container

Could not determine existing codeql version
Traceback (most recent call last):
  File "/usr/local/startup_scripts/setup.py", line 57, in <module>
    setup()
  File "/usr/local/startup_scripts/setup.py", line 39, in setup
    get_latest_codeql(args)
  File "/usr/local/startup_scripts/setup.py", line 44, in get_latest_codeql
    current_installed_version = codeql.get_current_local_version()
  File "/usr/local/startup_scripts/libs/codeql.py", line 74, in get_current_local_version
    exit(ERROR_EXECUTING_CODEQL)
NameError: name 'ERROR_EXECUTING_CODEQL' is not defined
Command Output:
[2021-03-08 23:32:49,409] INFO: Starting setup...

Error 1 executing from command.
Exiting...

dotnet-sdk-3.1 fails docker build

dotnet-sdk-3.1 fails docker build. Dotnet needs to be updated to dotnet-sdk-6.0 and use the new install scripts per:

https://learn.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#2004
https://learn.microsoft.com/en-us/dotnet/core/install/linux-scripted-manual#scripted-install

codeql-container/Dockerfile

Line 50 in 4632664

RUN apt-get install -y dotnet-sdk-3.1

Automated builds?

It seems that mcr.microsoft.com/cstsectools/codeql-container hasn't been updated since November 3rd, 2020 (CodeQL 2.3.2).

Would it be possible to have continuous builds of the codeql-container image? It looks like maybe that was the purpose of #1, but I'm not sure.

typo in Docker Hub description page

In How to Use this image tab.

docker run --rm --name codeql-container mcr.microsoft.com/codeql/codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS=

should be corrected to

docker run --rm --name codeql-container mcr.microsoft.com/cstsectools/codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS=

Get the latest commit instead of cloning everything

codeql-container/Dockerfile

Line 64 in d0e531a

# get the latest codeql queries and record the HEAD

The comment implies that only the latest commit is necessary to build the image. Since the CodeQL repository has an exceptionally large number of commits, it takes a significant amount of time to actually clone the project. In whole, there's some ~70 MB of files downloaded if a full clone takes place. A shallow clone of --depth 1 only is ~15 MB.

NameError: name 'ERROR_EXECUTING_CODEQL' is not defined

# docker run --rm --name codeql-container -e CODEQL_CLI_ARGS="resolve qlpacks"  mcr.microsoft.com/cstsectools/codeql-container

Could not determine existing codeql version
Traceback (most recent call last):
  File "/usr/local/startup_scripts/setup.py", line 57, in <module>
    setup()
  File "/usr/local/startup_scripts/setup.py", line 39, in setup
    get_latest_codeql(args)
  File "/usr/local/startup_scripts/setup.py", line 44, in get_latest_codeql
    current_installed_version = codeql.get_current_local_version()
  File "/usr/local/startup_scripts/libs/codeql.py", line 74, in get_current_local_version
    exit(ERROR_EXECUTING_CODEQL)
NameError: name 'ERROR_EXECUTING_CODEQL' is not defined
Error 1 executing from command.
Exiting...
Command Output: 
[2021-08-16 02:50:18,394] INFO: Starting setup...

NameError: name 'ERROR_EXECUTING_CODEQL' is not defined

The constants defined on the class CodeQL are not referenced correctly. For example:

codeql-container/container/libs/codeql.py

Line 43 in d0e531a

exit(ERROR_UNKNOWN_OS)

They should be accessed using the instance of the class via self.

Error: Could not determine existing codeql version

Hi There,

I am trying to scan my python code on my local workstation using the Codeql container However whenI tried to run this its keep sending me the error message mention the Could not determine existing codeql version. Can you please advise me if I am doing anything wrong here?

./analyze_security.bat C:\Temp\python\website\website C:\Temp\result python

Getting the image...
Using default tag: latest
latest: Pulling from cstsectools/codeql-container
Digest: sha256:87b399db8f3de7fc36e68f67f4d73fce8b073555411b51bd0c042b0dae0ada86
Status: Image is up to date for mcr.microsoft.com/cstsectools/codeql-container:latest
mcr.microsoft.com/cstsectools/codeql-container:latest
Pulled the container
Creating the codeQL database. This might take some time depending on the size of the project...
Could not determine existing codeql version
File "/usr/local/startup_scripts/setup.py", line 39, in setup
get_latest_codeql(args)
File "/usr/local/startup_scripts/setup.py", line 44, in get_latest_codeql
current_installed_version = codeql.get_current_local_version()
File "/usr/local/startup_scripts/libs/codeql.py", line 74, in get_current_local_version exit(ERROR_EXECUTING_CODEQL)NameError: name 'ERROR_EXECUTING_CODEQL' is not defined
Error 1 executing from command.Exiting...
Command Output:
[2021-11-18 01:33:59,959] INFO: Starting setup...

Failed creating the database

csharp database creation fails

Note due to #53 the image name is different:

creating a database fails

docker run --rm --name codeql-container -v `pwd`/backend/App:/opt/src -v `pwd`/tmp/:/opt/results -e CODEQL_CLI_ARGS="database create /opt/results/source_db -s /opt/src --language=csharp --overwrite" codeql

prints:

Initializing database at /opt/results/source_db.
Running build command: [/usr/local/codeql-home/codeql/csharp/tools/autobuild.sh]
A fatal error occurred: Exit status 1 from command: [/usr/local/codeql-home/codeql/csharp/tools/autobuild.sh]
Error 2 executing from command.
Exiting...
Command Output: 
[2023-05-22 06:50:12] [build-stdout] CodeQL C# autobuilder
[2023-05-22 06:50:12] [build-stdout] Working directory: /opt/src
[2023-05-22 06:50:12] [build-stdout] Attempting to build using .NET Core
[2023-05-22 06:50:12] [build-stdout] Running dotnet --info
[2023-05-22 06:50:12] [build-stderr] A fatal error occurred. The folder [/usr/share/dotnet/host/fxr] does not exist
[2023-05-22 06:50:12] [build-stdout] Exit code 131
[2023-05-22 06:50:12] [build-stdout] Attempting to build using MSBuild
[2023-05-22 06:50:12] [build-stdout] Running nuget restore /opt/src/App.csproj -DisableParallelProcessing
[2023-05-22 06:50:12] [build-stdout] Exit code 1: An error occurred trying to start process 'nuget' with working directory '/opt/src'. No such file or directory
[2023-05-22 06:50:12] [build-stdout] Attempting to download nuget.exe
[2023-05-22 06:50:33] [build-stdout] Successfully downloaded /opt/src/.nuget/nuget.exe
[2023-05-22 06:50:33] [build-stdout] Running mono /opt/src/.nuget/nuget.exe restore /opt/src/App.csproj -DisableParallelProcessing
[2023-05-22 06:50:33] [build-stdout] Exit code 1: An error occurred trying to start process 'mono' with working directory '/opt/src'. No such file or directory
[2023-05-22 06:50:33] [build-stdout] Running msbuild /t:restore /opt/src/App.csproj
[2023-05-22 06:50:33] [build-stdout] Exit code 1: An error occurred trying to start process 'msbuild' with working directory '/opt/src'. No such file or directory
[2023-05-22 06:50:33] [build-stdout] Running msbuild /opt/src/App.csproj /t:rebuild
[2023-05-22 06:50:33] [build-stdout] Exit code 1: An error occurred trying to start process 'msbuild' with working directory '/opt/src'. No such file or directory
[2023-05-22 06:50:33] [build-stdout] Attempting to locate build script
[2023-05-22 06:50:33] [build-stderr] Error: Could not auto-detect a suitable build method
[2023-05-22 06:50:33] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/usr/local/codeql-home/codeql/tools/linux64/preload_tracer, /usr/local/codeql-home/codeql/csharp/tools/autobuild.sh])

`CODEQL_GITHUB_URL` is never used

codeql-container/container/libs/codeql.py

Line 37 in d0e531a

download_url = f'https://github.com/github/codeql-cli-binaries/releases/download/{github_version.title}/codeql-linux64.zip'

It seems that the constant was meant to be used in place of the hardcoded URLs for downloading the CLI binaries. Simply replacing the base of hardcoded URL with the constant would leave the remaining portion of the URL hardcoded.

Unable to build due to missing exclude-dependency-queries.yml?

MacBook-Air:~ dave$ docker build https://github.com/microsoft/codeql-container.git#main -t codeql-container
Sending build context to Docker daemon  153.6kB
Step 1/21 : FROM ubuntu:20.04 AS codeql_base
 ---> 26b77e58432b
Step 2/21 : LABEL maintainer="Github codeql team"
 ---> Using cache
 ---> af7ff04e6772
Step 3/21 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> ee93c15db382
Step 4/21 : RUN apt-get update &&     apt-get upgrade -y &&     apt-get install -y --no-install-recommends     	software-properties-common     	vim     	curl     	wget     	git     	build-essential     	unzip     	apt-transport-https         python3.8     	python3-venv     	python3-pip     	python3-setuptools         python3-dev     	gnupg     	g++     	make     	gcc     	apt-utils         rsync     	file         dos2unix     	gettext &&         apt-get clean &&         ln -s /usr/bin/python3.8 /usr/bin/python &&         ln -s /usr/bin/pip3 /usr/bin/pip
 ---> Using cache
 ---> 7c1ae9f5efde
Step 5/21 : RUN cd /tmp &&     wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb &&     dpkg -i packages-microsoft-prod.deb &&     apt-get update;     apt-get install -y apt-transport-https &&     apt-get update &&     rm packages-microsoft-prod.deb
 ---> Using cache
 ---> faab64de08c3
Step 6/21 : RUN apt-get install -y dotnet-sdk-3.1
 ---> Using cache
 ---> 989aa1946455
Step 7/21 : RUN mkdir -p /usr/local/startup_scripts
 ---> Using cache
 ---> 0a37e643e2a6
Step 8/21 : RUN ls -al /usr/local/startup_scripts
 ---> Using cache
 ---> 00d7a0e25068
Step 9/21 : COPY container /usr/local/startup_scripts/
 ---> Using cache
 ---> 3ac1fd656165
Step 10/21 : RUN pip3 install --upgrade pip     && pip3 install -r /usr/local/startup_scripts/requirements.txt
 ---> Using cache
 ---> a60e576a2e88
Step 11/21 : ENV CODEQL_HOME /usr/local/codeql-home
 ---> Using cache
 ---> db67f9847013
Step 12/21 : RUN python3 /usr/local/startup_scripts/get-latest-codeql-version.py > /tmp/codeql_version
 ---> Using cache
 ---> 3887946b1e77
Step 13/21 : RUN mkdir -p ${CODEQL_HOME}     ${CODEQL_HOME}/codeql-repo     ${CODEQL_HOME}/codeql-go-repo     /opt/codeql
 ---> Using cache
 ---> 031a4cb13a05
Step 14/21 : RUN git clone https://github.com/github/codeql ${CODEQL_HOME}/codeql-repo &&     git --git-dir ${CODEQL_HOME}/codeql-repo/.git log --pretty=reference -1 > /opt/codeql/codeql-repo-last-commit
 ---> Using cache
 ---> 3f04a8f47768
Step 15/21 : RUN git clone https://github.com/github/codeql-go ${CODEQL_HOME}/codeql-go-repo &&     git --git-dir ${CODEQL_HOME}/codeql-go-repo/.git log --pretty=reference -1 > /opt/codeql/codeql-go-repo-last-commit
 ---> Using cache
 ---> a9c965608fb2
Step 16/21 : RUN CODEQL_VERSION=$(cat /tmp/codeql_version) &&     wget -q https://github.com/github/codeql-cli-binaries/releases/download/${CODEQL_VERSION}/codeql-linux64.zip -O /tmp/codeql_linux.zip &&     unzip /tmp/codeql_linux.zip -d ${CODEQL_HOME} &&     rm /tmp/codeql_linux.zip
 ---> Using cache
 ---> 439baa2d827d
Step 17/21 : ENV PATH="${CODEQL_HOME}/codeql:${PATH}"
 ---> Using cache
 ---> 2c8b0fada471
Step 18/21 : RUN codeql query compile --threads=0 ${CODEQL_HOME}/codeql-repo/*/ql/src/codeql-suites/*.qls
 ---> Running in 8a3742f74fe2
A fatal error occurred: /usr/local/codeql-home/codeql-repo/csharp/ql/src/codeql-suites/exclude-dependency-queries.yml which is referenced from /usr/local/codeql-home/codeql-repo/csharp/ql/src/codeql-suites/csharp-code-scanning.qls cannot be parsed as a query suite: java.io.FileNotFoundException: /usr/local/codeql-home/codeql-repo/csharp/ql/src/codeql-suites/exclude-dependency-queries.yml (No such file or directory)
The command '/bin/sh -c codeql query compile --threads=0 ${CODEQL_HOME}/codeql-repo/*/ql/src/codeql-suites/*.qls' returned a non-zero code: 2

Docker: Example command

I think the ordering of the docker run command in the README needs to be fixed. At least on Windows, if I have the image name before the environment vars, they don't seem to get passed into the container:

PS D:\temp\codeql-container> docker run --rm codeql:latest -v D:\ReferenceSource\libpng:/opt/src -v d:\temp\codeql-container\results:/opt/results --env CODEQL_CLI_ARGS="database create --language=cpp /opt/src/source_db2"
[2020-07-20 15:09:07,152] INFO: No valid argument passed in for codeql-cli, nothing to do. To perform some task, please set the CODEQL_CLI_ARGS environment variable to a valid argument...

PS D:\temp\codeql-container> docker run --rm -v D:\ReferenceSource\libpng:/opt/src -v d:\temp\codeql-container\results:/opt/results --env CODEQL_CLI_ARGS="database create --language=cpp /opt/src/source_db3" codeql:latest
Initializing database at /opt/src/source_db3.

C# query compile failed

looks like at least one query faild in the compile step. Maybe we can accept failing queries instead of failing the build

Compiling query plan for /usr/local/codeql-home/codeql-repo/csharp/ql/src/Security Features/CWE-502/UnsafeDeserializationUntrustedInput.ql.
The command '/bin/sh -c codeql query compile ${CODEQL_HOME}/codeql-repo/csharp/ql/src/codeql-suites/*.qls --additional-packs=.' returned a non-zero code: 137

CSharp and Java Queries Not Cached / Precompiled

Starting a few weeks ago, CSharp and Java codeql analysis are taking extremely long... around 60 min, where the same analysis was taking 2 min prior. It looks as though the queries are not hitting the precompiled cache and are compiling on the fly with the following output:

[1/174] Compiled /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToGCCollect.ql.
Compiling query plan for /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.
Resolving imports for /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.
Checking QL for /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.
Optimizing /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.

Previously the output showed a cache hit when executing much faster:

[1/173] Found in cache: /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToGCCollect.ql.
Compiling query plan for /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.
Resolving imports for /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.
Compilation cache hit for /usr/local/codeql-home/codeql-repo/csharp/ql/src/API Abuse/CallToObsoleteMethod.ql.

I don't have any local changes nor do I see any changes to this repository in the last few weeks to this repository. Perhaps something in the weekly build process is failing to precompile all languages. Python and Node seem to be working fine on latest container showing cache hits on all queries.

Outdated scripts reference deprecated `--format=sarifv2` causing scripts to fail

The Python scripts in project contain references to a deprecated format.

codeql-container/scripts/unix/analyze_security.sh

Line 61 in 073695d

docker run --rm --name codeql-container -v ${inputfile}:/opt/src -v ${outputfile}:/opt/results -e CODEQL_CLI_ARGS=database\ analyze\ /opt/results/source_db\ --format=sarifv2\ --output=/opt/results/issues.sarif\ ${language}-security-and-quality.qls mcr.microsoft.com/cstsectools/codeql-container

Error:

A fatal error occurred: Support for SARIF v2.0.0 (Committee Specification Draft 1) was removed in v2.8.2 of the CodeQL CLI. If you need this functionality, please file a public issue in https://github.com/github/codeql-cli-binaries, or open a private ticket with GitHub Support and request an escalation to engineering.
Error 2 executing from command.

Outcome:
When this happens the issues.sarif is not created.

Workaround:
Replace with --format=sarif-latest

container tags by version

it would be great to have container tags by version, and not only "latest" as this could break existing flows if the "latest" tag is updated.

using this manifest: https://mcr.microsoft.com/v2/cstsectools/codeql-container/tags/list
I can see that only "latest" exists.

Version detect fails

Could not determine existing codeql version
Traceback (most recent call last):
  File "/usr/local/startup_scripts/setup.py", line 57, in <module>
    setup()
  File "/usr/local/startup_scripts/setup.py", line 39, in setup
    get_latest_codeql(args)
  File "/usr/local/startup_scripts/setup.py", line 44, in get_latest_codeql
    current_installed_version = codeql.get_current_local_version()
  File "/usr/local/startup_scripts/libs/codeql.py", line 74, in get_current_local_version
    exit(ERROR_EXECUTING_CODEQL)
NameError: name 'ERROR_EXECUTING_CODEQL' is not defined
Error 1 executing from command.
Exiting...
Command Output: 
[2020-12-01 12:15:08,245] INFO: Starting setup...

The 2.3.2 and 2.4.0 tags of the base codeql util reports version differently than expected:

$ docker run -it --entrypoint /bin/bash mcr.microsoft.com/cstsectools/codeql-container:latest
root@64e686e098fd:/# /usr/local/codeql-home/codeql/codeql version
CodeQL command-line toolchain release 2.3.2.
Copyright (C) 2019-2020 GitHub, Inc.
Unpacked in: /usr/local/codeql-home/codeql
   Analysis results depend critically on separately distributed query and
   extractor modules. To list modules that are visible to the toolchain,
   use 'codeql resolve qlpacks' and 'codeql resolve languages'.

root@6f4812e540a7:/# /usr/local/codeql-home/codeql/codeql version
CodeQL command-line toolchain release 2.4.0.
Copyright (C) 2019-2020 GitHub, Inc.
Unpacked in: /usr/local/codeql-home/codeql
   Analysis results depend critically on separately distributed query and
   extractor modules. To list modules that are visible to the toolchain,
   use 'codeql resolve qlpacks' and 'codeql resolve languages'.

Errors have hard-to-read output

When an error occurs, the output comes back in a hard-to-read form, e.g.:

PS D:\temp\codeql-container> docker run --rm -v D:\ReferenceSource\libpng:/opt/src -v d:\temp\codeql-container\results:/opt/results --env CODEQL_CLI_ARGS="database create --language=cpp /opt/src/source_db3" codeql:latest
Initializing database at /opt/src/source_db3.
Running command [/usr/local/codeql-home/codeql/cpp/tools/autobuild.sh] in /.
A fatal error occurred: Exit status 1 from command: [/usr/local/codeql-home/codeql/cpp/tools/autobuild.sh]
Error 2,b"[2020-07-20 15:10:14] [build-err] + /usr/local/codeql-home/codeql/cpp/tools/lgtm-scripts/cpp/detect_source_root\n[2020-07-20 15:10:14] [build] detect_source_root: Build will be attempted from '.'\n[2020-07-20 15:10:14] [build] even though no build system was found there.\n[2020-07-20 15:10:14] [build-err] + '[' -L _lgtm_detected_source_root ']'\n[2020-07-20 15:10:14] [build-err] ++ readlink _lgtm_detected_source_root\n[2020-07-20 15:10:14] [build-err] + cd .\n[2020-07-20 15:10:14] [build-err] + export CXXFLAGS=-fpermissive\n[2020-07-20 15:10:14] [build-err] + CXXFLAGS=-fpermissive\n[2020-07-20 15:10:14] [build-err] + configure_prefix_opt=\n[2020-07-20 15:10:14] [build-err] + '[' -f configure.ac ']'\n[2020-07-20 15:10:14] [build-err] + '[' -f configure.in ']'\n[2020-07-20 15:10:14] [build-err] + '[' -f CMakeLists.txt ']'\n[2020-07-20 15:10:14] [build-err] + '[' -f meson.build ']'\n[2020-07-20 15:10:14] [build-err] + try_configure\n[2020-07-20 15:10:14] [build-err] + build_dir=.\n[2020-07-20 15:10:14] [build-err] + root_dir=.\n[2020-07-20 15:10:14] [build-err] + grep -q 'AC_MSG_.* configure in a separate.* directory' configure.in configure.ac\n[2020-07-20 15:10:14] [build-err] + for configure in configure configure.gnu\n[2020-07-20 15:10:14] [build-err] + '[' -x configure ']'\n[2020-07-20 15:10:14] [build-err] + for configure in configure configure.gnu\n[2020-07-20 15:10:14] [build-err] + '[' -x configure.gnu ']'\n[2020-07-20 15:10:14] [build-err] + rm -rf _lgtm_build_dir\n[2020-07-20 15:10:14] [build-err] + for bootstrap in bootstrap.sh bootstrap autogen.sh\n[2020-07-20 15:10:14] [build-err] + '[' -x bootstrap.sh ']'\n[2020-07-20 15:10:14] [build-err] + '[' -f bootstrap.sh ']'\n[2020-07-20 15:10:14] [build-err] + for bootstrap in bootstrap.sh bootstrap autogen.sh\n[2020-07-20 15:10:14] [build-err] + '[' -x bootstrap ']'\n[2020-07-20 15:10:14] [build-err] + '[' -f bootstrap ']'\n[2020-07-20 15:10:14] [build-err] + for bootstrap in bootstrap.sh bootstrap autogen.sh\n
...

I'd suggest decoding and printing raw to the console.