globocom / huskyci Goto Github PK
View Code? Open in Web Editor NEWPerforming security tests inside your CI
Home Page: https://huskyci.opensource.globo.com
License: BSD 3-Clause "New" or "Revised" License
Performing security tests inside your CI
Home Page: https://huskyci.opensource.globo.com
License: BSD 3-Clause "New" or "Revised" License
Although an analysis ran perfectly fine, some status field are being written as "timedout" in MongoDB.
gas
tool has changed its name into gosec
. It would be a good idea to do a refactor.
Well, sometimes it happens..
func (d Docker) ReadOutput() (string, error) {
dockerHost := os.Getenv("DOCKER_HOST")
URL := "http://" + dockerHost + "/v1.24/containers/" + d.CID + "/logs?stdout=1"
resp, err := http.Get(URL)
if err != nil {
return "", err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return "", err
}
return string(body), err
}
URL := "http://" + dockerHost + "/v1.24/containers/" + d.CID + "/logs?stdout=1"
is being used but it would be nice to check if
URL := "http://" + dockerHost + "/v1.24/containers/" + d.CID + "/logs?stderr=1"
is also returns something!
The following error occurred when running Huksy on https://github.com/globocom/snap-plugin-collector-docker.git
repository:
Unmarshall error (gas.go): json: cannot unmarshal array into Go value of type analysis.GasOutput [0]
cStatus
is running
forever
The following commands may need to use git private and pub ssh keys to clone private repositories:
git clone %GIT_REPO% code --quiet && cd code && enry --json | tr -d '\r\n'
cd src; git clone %GIT_REPO% code --quiet && cd code && /go/bin/gas -quiet -fmt=json -log=log.txt -out=results.json ./... 2> /dev/null ; jq -j -M -c . results.json
Husky could rescan (enry) a repository to identify new languages/files after a specific number sent by the user.
Requests could use an "huskyci token" to avoid malicious users from abusing the API.
Although it would be nice avoiding a new Enry scan every time Husky starts an analysis, this might be necessary to guarantee that any vulnerability is not detected if a new language is inserted.
Example:
A new config.sh
is added into the repository that contains hardcoded passwords. Enry may not detect this file for the futures 9 scans in the worst case scenario.
Create a new standard for all error messages to be used by Husky as suggested by @nettoclaudio:
I would returned the echo.NewHTTPError(http.StatusNotFound, "analysis not found") instead that. I think that's the standardized by Echo web framework.
Ref.: https://echo.labstack.com/guide/error-handling
MongoDB is receiving a lot of connections because HuskyCI code is using the following code very often:
session, err := db.Connect()
As Docker API must be configured to accept only requests form HuskyCI via HTTPS and Client Auth, dockers/api.go
must be refactored.
After installing the environment via make install
, the following error is being found:
$ docker logs huskyCIAPI
server.go:10:2: cannot find package "github.com/globocom/glbgelf" in any of:
/go/src/github.com/globocom/huskyci/vendor/github.com/globocom/glbgelf (vendor tree)
/usr/local/go/src/github.com/globocom/glbgelf (from $GOROOT)
/go/src/github.com/globocom/glbgelf (from $GOPATH)
Let's now check JS code! o/
@carlosljr : "It seems that you need to wait one more retryTick to update status of analysisFinished to true, right? Why not update in the moment that the CStatus of all containers are detected as finished? In that case, all the logic from lines 164 to 191 are passed to line 205."
Husky must have a client with the following features:
When cloning an empty repository, Husky must insert into MongoDB all information regarding the errors found.
The repository example https://github.com/vulnaas/vulnaas-api.git
, at this moment, has only:
Who doesn't love huskies? ๐ถ
We must check the Copyright of the image!
Implement HTTPS!
Images like enry
and gas
have ~800 MB. Check if it is possible to create much smaller images.
The images used in Docker to start the containers with security tests is downloaded once from Vagrant script. However, image management must be done on the API. If the images are removed from Docker, the security tests will fail.
The HuskyCI API needs to know if images that deal with security tests exist and, if not, get from the correspondent registry. After that, the logic follows the current procedure.
Thanks for reporting this @carlosljr !
git clone %GIT_REPO% code --quiet 2> /dev/null
Should be something like this:
git clone -b %GIT_BRANCH% %GIT_REPO% code --quiet 2> /dev/null
Gitlab CI has the default Env Var CI_COMMIT_REF_NAME
for this (https://docs.gitlab.com/ee/ci/variables/#predefined-variables-environment-variables).
It would be a good idea to validate this and find alternatives for Github.
It would be nice if HuskyCI checks docker images (huskyci/enry, huskyci/gas, etc) in docker API before starting.
When starting an analysis on a "big" repository such as golang/go.git, the following error is being shown:
Unmarshall error: invalid character '\x01' in string literal
That's the error being found in MongoDB when performing analysis on Python projects!
This specific function may grow in time and it would be a good idea to store it in a different file rather then analysis.go
.
The following repository was stated as not a git repository:
[email protected]:example/example.git
The commands are wrong! Create new ones and update README.md
Some good examples:
retryTick
timeout
limitEnryScan
This file must not store any sensitive data.
The map interface being used to query MongoDB by an analysis ID is not working:
repositoryQuery := map[string]interface{}{"_id": analysis.ID}
When building Husky client, the following error is being received when trying to GET an analysis:
Error during GET to Husky API: Get http://localhost:9999/husky/g3i2Yqrwv1hQfejdQehSMIOtcmOZsJCJ: 302 response missing Location header
The problem might be at return c.JSON(http.StatusFound, analysisResult)
from analysis.go
. A different status code like StatusOK would solve this problem.
Example:
Cloning into 'code'...
remote: Counting objects: 1620, done.
remote: Compressing objects: 100% (1141/1141), done.
remote: Total 1620 (delta 420), reused 1594 (delta 398), pack-reused 0
Receiving objects: 100% (1620/1620), 3.69 MiB | 2.15 MiB/s, done.
Resolving deltas: 100% (420/420), done.
{"Dockerfile":["Dockerfile"],"Go":["api/health.go","api/health_test.go","api/mock.go","api/scan.go","api/scan_test.go","api/server.go","cmd/root.go","cmd/server/server.go","cmd/server/server_test.go","cmd/worker/worker.go","cmd/worker/worker_test.go","db/mock.go","db/mongodb/mongo.go","db/mongodb/mongo_test.go","db/storage.go","main.go","queue/mock.go","queue/queue.go","scan/clair.go","scan/mock.go","scan/scan.go","scan/scheduler/default.go","scan/scheduler/default_test.go","scan/scheduler/mock.go","scan/scheduler/scheduler.go","scan/worker/default.go","scan/worker/default_test.go"],"Makefile":["Makefile"]}
Expected:
{"Dockerfile":["Dockerfile"],"Go":["api/health.go","api/health_test.go","api/mock.go","api/scan.go","api/scan_test.go","api/server.go","cmd/root.go","cmd/server/server.go","cmd/server/server_test.go","cmd/worker/worker.go","cmd/worker/worker_test.go","db/mock.go","db/mongodb/mongo.go","db/mongodb/mongo_test.go","db/storage.go","main.go","queue/mock.go","queue/queue.go","scan/clair.go","scan/mock.go","scan/scan.go","scan/scheduler/default.go","scan/scheduler/default_test.go","scan/scheduler/mock.go","scan/scheduler/scheduler.go","scan/worker/default.go","scan/worker/default_test.go"],"Makefile":["Makefile"]}
DockerHostsConfig
struct so that it stores:Addresses
Protocol
DockerAPIVersion
@mdjunior : "The DockerAPI port that will be the same on all hosts?
Maybe it was better if the data structure allowed a list of hosts, where each host would be a structure with Address and Port (and eventually other things, such as certificates and keys)."
// DockerHostsConfig represents Docker Hosts configuration.
type DockerHostsConfig struct {
Addresses []string
DockerAPIPort int
@mdjunior : "The http protocol and docker version is hardcoded. Maybe it was better if this information was entered in the DockerHosts data structure in context/config.go."
// HealthCheckAPI returns true if a 200 status code is received or false otherwise.
func HealthCheckAPI(dockerHost string) error {
URL := fmt.Sprintf("http://%s/v1.24/version", dockerHost)
Repository's document must store only the Object-ID from a securityTest and not the entire document.
This issue may help during an eventually future securityTest update.
Security tests configs could have cmds to git clone inside container with or without using SSH keys:
For example cmdSSHAuth
and cmdNonAuth
:
enry:
name: enry
image: huskyci/enry
cmdSSHAuth: |+
mkdir -p ~/.ssh &&
echo 'GIT_PRIVATE_SSH_KEY' > ~/.ssh/huskyci_id_rsa &&
chmod 600 ~/.ssh/huskyci_id_rsa &&
echo "IdentityFile ~/.ssh/huskyci_id_rsa" >> /etc/ssh/ssh_config &&
echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config &&
git clone -b %GIT_BRANCH% --single-branch %GIT_REPO% code --quiet 2> /tmp/errorGitCloneEnry
if [ $? -eq 0 ]; then
cd code
enry --json | tr -d '\r\n'
else
echo "ERROR_CLONING"
cat /tmp/errorGitCloneEnry
fi
cmdNonAuth: |+
git clone -b %GIT_BRANCH% --single-branch %GIT_REPO% code --quiet 2> /tmp/errorGitCloneEnry
if [ $? -eq 0 ]; then
cd code
enry --json | tr -d '\r\n'
else
echo "ERROR_CLONING"
cat /tmp/errorGitCloneEnry
fi
language: Generic
default: true
timeOutInSeconds: 60
For statistics purposes, it would be a good idea to implement the field numAnalysis
into Repository
.
An increment function is also needed when updating it into MongoDB.
type Repository struct {
ID bson.ObjectId `bson:"_id,omitempty"`
URL string `bson:"URL" json:"repositoryURL"`
SecurityTests []SecurityTest `bson:"securityTests" json:"securityTests"`
SecurityTestName []string `bson:"securityTestName,omitempty" json:"securityTestName"`
VM string `bson:"VM" json:"vm"`
CreatedAt time.Time `bson:"createdAt" json:"createdAt"`
DeletedAt time.Time `bson:"deletedAt" json:"deletedAt"`
Languages []Language `bson:"languages" json:"languages"`
}
The collection must store host, port, certificate and other information.
Remove all fmt.Println() messages when errors occur and add a log system into Husky.
Some ideas:
@carlosljr , is glbgelf going to be added as an open source project?
Husky must not start before checking the following services:
enrySecurityTestQuery := map[string]interface{}{"name": "enry"}
enrySecurityTestResult, err := FindOneDBSecurityTest(enrySecurityTestQuery)
if err != nil {
fmt.Println("Error finding enry securityTest:", err)
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.