bbva / deeptracy

The Security Dependency Orchestrator Service

CAVEAT EMPTOR: This project is not being maintained anymore. Use with caution!

Deeptracy

Deeptracy scans your project dependencies to spot vulnerabilities.

Is a meta tool to analyze the security issues in third party libraries used in your project.

We have created this project to simplify this process so you can focus only in the important: your project.

Deeptracy can choose the most suitable security tools for each languages and notify the spotted vulnerabilities in the project dependencies.

You can learn more about Deeptracy in the official documentation.

This project is distributed under Apache License.

deeptracy's Issues

API versioning on header

We could set the requested API version on Header in order to preserve the same path across different API versions

Postman with HTTP calls to create objects

Firebase push notifications

Add firebase support to send push notifications when a scan finishes

Create deploy template for rancher

Provide a rancher template to deploy the complete deeptracy solution

Limit scans by time

Scans should be limited by time.

A single project can execute a scan only 1 per 15 mins (configurable)

See: https://flask-limiter.readthedocs.io/en/stable/

User documentation

Write a complete user documentation and publish it to read the docs.

This documentation should include API, usage and deployment documentation

Integrating Stash with Deeptracy

Hi, trhanks for distribute this software seems pretty interesting, I found that in:
https://www.bbva.com/es/gestion-vulnerabilidades-dependencias-entornos-cicd-herramientas-open-
source/

I've configured a my env with the following docker-compose:

version: '3'

services:

  postgres:
    image: postgres:9.6-alpine
    environment:
      - POSTGRES_PASSWORD=postgres
    ports:
      - 5433:5433
    command: -p 5433

  redis:
    image: redis:3-alpine
    ports:
      - 6380:6380

  deeptracy:
    image: bbvalabs/deeptracy
    depends_on:
      - redis
      - postgres
    environment:
      - BROKER_URI=redis://redis:6379
      - DATABASE_URI=postgresql://postgres:postgres@postgres:5433/deeptracy
      - POSTGRES_URI=postgresql://postgres:postgres@postgres:5433
      - SHARED_VOLUME_PATH=/tmp/deeptracy
      - LOCAL_PRIVATE_KEY_FILE=/root/.ssh/id_rsa
      - PATTON_URI=http://0.0.0.0:8000
      # - EMAIL_SMTP_SERVER=xxx.xxx.xxx
      # - EMAIL_SMTP_PORT=xxx
      # - [email protected]
      # - EMAIL_SMTP_PASSWORD=xxxxx
      # - [email protected]
    ports:
      - 8000:8000
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp:/tmp
      - ./private_key:/root/.ssh/
    privileged: true
    command: ["./init_patton_db.sh"]

  patton-server:
    image: bbvalabs/patton-server
    environment:
      - WORKERS=1
      - BACKLOG=512
      - LISTEN_PORT=9000
      - POSTGRES_HOST=postgres
      - POSTGRES_PORT=5433
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=patton
    ports:
      - 9000:8000
    depends_on:
      - postgres

  deeptracy-api:
    image: bbvalabs/deeptracy-api
    depends_on:
      - redis
      - postgres
      - deeptracy
    ports:
      - 8081:8081
    environment:
      - BROKER_URI=redis://redis:6380
      - DATABASE_URI=postgresql://postgres:postgres@postgres:5433/deeptracy
      - SERVER_ADDRESS=0.0.0.0:8081
      - GUNICORN_WORKERS=1
      - LOG_LEVEL=INFO
    command: ["./wait-for-it.sh", "postgres:5433", "--", "/opt/deeptracy/run.sh"]

Also I've configured a webhook on my git repository (stash), I've configured as follows:

But when the hook is launched I'm seeing a "BAD REQUEST" in deeptracy-api logs without extra information:

deeptracy_1      | [2018-09-04 12:47:01,432: INFO/MainProcess] Connected to redis://redis:6379//
deeptracy_1      | [2018-09-04 12:47:01,453: INFO/MainProcess] mingle: searching for neighbors
deeptracy_1      | [2018-09-04 12:47:02,496: INFO/MainProcess] mingle: all alone
deeptracy_1      | [2018-09-04 12:47:02,520: INFO/MainProcess] celery@fb67cffc785d ready.
    > nvdcve-2.0-2003.xml.gz: 440kB [00:00, 575kB/s]                           00<00:00, 331kB/s]
    > nvdcve-2.0-2005.xml.gz: 1.35MB [00:01, 1.27MB/s]                           <00:00, 607kB/s]
    > nvdcve-2.0-2015.xml.gz: 2.35MB [00:01, 1.74MB/s]                            1<00:00, 1.01MB/s]
    > nvdcve-2.0-2009.xml.gz: 2.15MB [00:01, 1.79MB/s]                           01<00:00, 895kB/s]
    > nvdcve-2.0-2011.xml.gz: 6.32MB [00:01, 3.68MB/s]                            :01<00:00, 2.61MB/s]
    > nvdcve-2.0-2016.xml.gz: 2.75MB [00:03, 865kB/s]                            0:03<00:00, 975kB/s]
    > nvdcve-2.0-2008.xml.gz: 2.22MB [00:02, 836kB/s]                            0:02<00:00, 960kB/s]
    > nvdcve-2.0-2014.xml.gz: 2.79MB [00:01, 2.10MB/s]                            1<00:00, 1.16MB/s]
    > official-cpe-dictionary_v2.3.xml.gz: 2.57MB [00:01, 1.93MB/s]                            1<00:00, 1.13MB/s]
    > nvdcve-2.0-2010.xml.gz: 2.90MB [00:01, 1.30MB/s]                           <00:00, 960kB/s]
    > nvdcve-2.0-2018.xml.gz: 2.89MB [00:01, 2.22MB/s]                            1<00:00, 1.14MB/s]
    > nvdcve-2.0-2006.xml.gz: 2.11MB [00:01, 1.96MB/s]                           <00:00, 774kB/s]
deeptracy-api_1  | [2018-09-04 12:55:20,371 deeptracy    INFO    ] 192.168.151.132 POST http /api/1/webhook/? 400 BAD REQUEST

Could you help me with that ?

Change behave targets name to at

Change targest names in Makefile from behave to at

Add sentry

Add test with a broken plugin to check inconsistent states

Now we are storing intermediate state like scan.analysis_countand scan.analysis_done.
If celery brokes, or a plugin fails this should reflect the incomplete state.

Now it's difficult because locally the environment is manually loaded before running behave.
We should normalize local and travis behaves to ease that and add this test.

Add proper logs to applications

All pieces has poor logs and low info on execution and development mode. We need to configure loggers with proper formatting, allowing some configuration to be taken from the environment (like the log level and destination)

Improve CI flow auto generate dev deeptracy-core package

When upload develop changes in api or workers, if this changes depends on deeptracy-core, get the master packages on pip repository, and fails. The task is generate a dev pip packages for pipelines, to avoid this fails.

Add codecov to all projects

With visibility to our code coverage we are going to be more encouraged to rise that coverage (right now is poor) on every pull request.

Add scan properties

Scan need to have the following properties:

total_packages
total_vulnerabilties

jwt.exceptions.InvalidSignatureError: Signature verification failed

The master output this warning when a browser connects to it:

deeptracy-buildbot_1  |         Traceback (most recent call last):
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/web/server.py", line 481, in getSession
deeptracy-buildbot_1  |             session = self.site.getSession(sessionCookie)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/www/service.py", line 174, in getSession
deeptracy-buildbot_1  |             return BuildbotSession(self, uid)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/www/service.py", line 74, in __init__
deeptracy-buildbot_1  |             self._fromToken(token)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/www/service.py", line 88, in _fromToken
deeptracy-buildbot_1  |             log.err(e, "while decoding JWT session")
deeptracy-buildbot_1  |         --- <exception caught here> ---
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/www/service.py", line 84, in _fromToken
deeptracy-buildbot_1  |             SESSION_SECRET_ALGORITHM])
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/jwt/api_jwt.py", line 93, in decode
deeptracy-buildbot_1  |             jwt, key=key, algorithms=algorithms, options=options, **kwargs
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/jwt/api_jws.py", line 157, in decode
deeptracy-buildbot_1  |             key, algorithms)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/jwt/api_jws.py", line 224, in _verify_signature
deeptracy-buildbot_1  |             raise InvalidSignatureError('Signature verification failed')
deeptracy-buildbot_1  |         jwt.exceptions.InvalidSignatureError: Signature verification failed
deeptracy-buildbot_1  | 

Related to #96 .

Remove docker-compose inside behave environment

Now behave is configured to launch docker-compose (test/acceptance/environment.py) after launching the tests. This dependency should be removed. The environment should be provided by externals sources (maybe in the make file or in travis).

Add support to open jira tickets

When a scan is finished, we want to open a issue directly in Jira with the vulnerability list.

To be able to get this feature we need more info at project creation time, mostly the credentials to be able to open the issues.

Realted #28

Improve docker-hub descriptions

The two images we are building in docker hub needs info, description, links and such.

Problem starting analysis

Hi everyone,

I've just followed the installation docs of the new version of deeptracy (on a fresh install of ubuntu server 18.04), I prepared a demo repo with this configuration (.deeptracy.yml):

projects:
    TestCi:
        type: deeptracy-mvn:3.5-jdk-8
        strategy: mvn_dependencytree
        unimportant: false
        config:
            path: src/main

But when I ran this call:

curl --data '{"repository": "http://stash/stash/scm/aeas/testci.git", "commit": "17e74a2a72cd15539f04d1ab888a4a98c13a1b65"}' -XPOST  http://localhost:8088/analysis/ -H "Content-Type: application/json"

I saw in the logs this error:

deeptracy-server_1    | 172.18.0.1 - - [20/Sep/2018 17:56:39] "POST /analysis/ HTTP/1.1" 200 46
deeptracy-worker_1    | [2018-09-20 17:56:39,022: INFO/MainProcess] Received task: deeptracy.tasks.request_extraction[78e1b7b4-81d6-4e96-b356-54c2450226e1]  
deeptracy-buildbot_1  | 2018-09-20 17:56:39+0000 [_GenericHTTPChannelProtocol,1,172.18.0.5] WARNING: change source is using deprecated self.master.addChange method; this method will disappear in Buildbot-1.0.0
deeptracy-buildbot_1  | 2018-09-20 17:56:39+0000 [-] added change with revision 17e74a2a72cd15539f04d1ab888a4a98c13a1b65 to database
deeptracy-buildbot_1  | 2018-09-20 17:56:39+0000 [-] injected change Change(revision='17e74a2a72cd15539f04d1ab888a4a98c13a1b65', who='deeptracy', branch=None, comments='', when=1537466199, category=None, project='http://stash/stash/scm/aeas/testci.git', repository='http://stash/stash/scm/aeas/testci.git', codebase='')
deeptracy-worker_1    | /usr/local/lib/python3.6/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is
deeptracy-worker_1    | absolutely not recommended!
deeptracy-worker_1    | 
deeptracy-worker_1    | Please specify a different user using the --uid option.
deeptracy-worker_1    | 
deeptracy-worker_1    | User information: uid=0 euid=0 gid=0 egid=0
deeptracy-worker_1    | 
deeptracy-worker_1    |   uid=uid, euid=euid, gid=gid, egid=egid,
deeptracy-worker_1    | [2018-09-20 17:56:39,159: INFO/ForkPoolWorker-3] Task deeptracy.tasks.request_extraction[78e1b7b4-81d6-4e96-b356-54c2450226e1] succeeded in 0.1342389319997892s: None
deeptracy-buildbot_1  | 2018-09-20 17:56:40+0000 [-] added buildset 3 to database
deeptracy-buildbot_1  | 2018-09-20 17:56:40+0000 [-] starting build <Build launch number:None results:success> using worker <LatentWorkerForBuilder builder='launch' worker='docker-8' state=AVAILABLE>
deeptracy-buildbot_1  | 2018-09-20 17:56:40+0000 [-] <Build launch number:None results:success>.startBuild
deeptracy-buildbot_1  | 2018-09-20 17:56:40+0000 [-] acquireLocks(worker <WasherDockerLatentWorker 'docker-8'>, locks [])
deeptracy-buildbot_1  | 2018-09-20 17:56:40+0000 [-] substantiating worker <LatentWorkerForBuilder builder='launch' worker='docker-8' state=DETACHED>
deeptracy-buildbot_1  | 2018-09-20 17:56:40+0000 [-] Container created, Id: 241fad...
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [-] Container started
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [-] docker VM 241fad: b'2018-09-20T17:56:41+0000 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 18.4.0 (/usr/local/bin/python 3.6.5) starting up.\n2018-09-20T17:56:41+0000 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.'
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [-] docker VM 241fad: b'2018-09-20T17:56:41+0000 [-] Starting Worker -- version: 1970.01.01'
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [-] docker VM 241fad: b'2018-09-20T17:56:41+0000 [-] recording hostname in twistd.hostname'
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [-] docker VM 241fad: b'2018-09-20T17:56:41+0000 [buildbot_worker.pb.BotFactory#info] Starting factory <buildbot_worker.pb.BotFactory object at 0x7f7c63e70b38>\n2018-09-20T17:56:41+0000 [-] Connecting to 172.17.0.1:9989'
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] worker 'docker-8' attaching from IPv4Address(type='TCP', host='172.18.0.1', port=42900)
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [-] docker VM 241fad: b'2018-09-20T17:56:41+0000 [HangCheckProtocol,client] message from master: attached'
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] Got workerinfo from 'docker-8'
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [-] bot attached
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [-] docker VM 241fad: b"2018-09-20T17:56:41+0000 [HangCheckProtocol,client] I have a leftover directory 'zope' that is not being used by the buildmaster: you can delete it now"
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] Worker docker-8 attached to launch
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] Worker docker-8 attached to analyze
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] Worker docker-8 attached to bootstrap
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] Worker docker-8 substantiated \o/
deeptracy-buildbot_1  | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] Firing docker-8 substantiation deferred with success
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] starting build <Build launch number:2 results:success>.. pinging the worker <LatentWorkerForBuilder builder='launch' worker='docker-8' state=BUILDING>
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] sending ping
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [Broker,4,172.18.0.1] ping finished: success
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] added buildset 4 to database
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] releaseLocks(Trigger(schedulerNames=['bootstrap'], set_properties={'virtual_builder_name': Property(project)})): []
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-]  step 'trigger' complete: success (None)
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-]  <Build launch number:2 results:success>: build finished
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] starting build <Build bootstrap number:None results:success> using worker <LatentWorkerForBuilder builder='bootstrap' worker='docker-0' state=AVAILABLE>
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] <Build bootstrap number:None results:success>.startBuild
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] releaseLocks(<WasherDockerLatentWorker 'docker-8'>): []
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] disconnecting old worker docker-8 now
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] waiting for worker to finish disconnecting
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [Broker,4,172.18.0.1] Worker.detached(docker-8)
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] Stopping container 241fad...
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [Broker,4,172.18.0.1] releaseLocks(<WasherDockerLatentWorker 'docker-8'>): []
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] acquireLocks(worker <WasherDockerLatentWorker 'docker-0'>, locks [])
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] substantiating worker <LatentWorkerForBuilder builder='bootstrap' worker='docker-0' state=DETACHED>
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] Container created, Id: 3d4c6e...
deeptracy-buildbot_1  | 2018-09-20 17:56:42+0000 [-] Container started
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] docker VM 3d4c6e: b'2018-09-20T17:56:43+0000 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 18.4.0 (/usr/local/bin/python 3.6.5) starting up.'
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] docker VM 3d4c6e: b'2018-09-20T17:56:43+0000 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.\n2018-09-20T17:56:43+0000 [-] Starting Worker -- version: 1970.01.01\n2018-09-20T17:56:43+0000 [-] recording hostname in twistd.hostname'
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] docker VM 3d4c6e: b'2018-09-20T17:56:43+0000 [buildbot_worker.pb.BotFactory#info] Starting factory <buildbot_worker.pb.BotFactory object at 0x7feac505dba8>'
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] docker VM 3d4c6e: b'2018-09-20T17:56:43+0000 [-] Connecting to 172.17.0.1:9989'
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] worker 'docker-0' attaching from IPv4Address(type='TCP', host='172.18.0.1', port=42904)
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] docker VM 3d4c6e: b'2018-09-20T17:56:43+0000 [HangCheckProtocol,client] message from master: attached'
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Got workerinfo from 'docker-0'
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] bot attached
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] docker VM 3d4c6e: b"2018-09-20T17:56:43+0000 [HangCheckProtocol,client] I have a leftover directory 'zope' that is not being used by the buildmaster: you can delete it now\n2018-09-20T17:56:43+0000 [HangCheckProtocol,client] I have a leftover directory 'sqlalchemy' that is not being used by the buildmaster: you can delete it now"
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Worker docker-0 attached to launch
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Worker docker-0 attached to analyze
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Worker docker-0 attached to bootstrap
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Worker docker-0 substantiated \o/
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Firing docker-0 substantiation deferred with success
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] starting build <Build bootstrap number:2 results:success>.. pinging the worker <LatentWorkerForBuilder builder='bootstrap' worker='docker-0' state=BUILDING>
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] sending ping
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] ping finished: success
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] releaseLocks(SetProperty('repopath', '/work')): []
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-]  step 'SetProperty' complete: success (None)
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] <RemoteShellCommand '['git', '--version']'>: RemoteCommand.run [4]
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] command '['git', '--version']' in dir '/work'
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] <RemoteShellCommand '['git', '--version']'> rc=0
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] <RemoteCommand 'stat' at 139999432269776>: RemoteCommand.run [5]
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] <RemoteCommand 'stat' at 139999432269776> rc=2
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] <RemoteCommand 'listdir' at 139999559733936>: RemoteCommand.run [6]
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] <RemoteCommand 'listdir' at 139999559733936> rc=0
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] No git repo present, making full clone
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] <RemoteShellCommand '['git', 'clone', 'http://stash/stash/scm/aeas/testci.git', '.']'>: RemoteCommand.run [7]
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] command '['git', 'clone', 'http://stash/stash/scm/aeas/testci.git', '.']' in dir '/work'
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] <RemoteShellCommand '['git', 'clone', 'http://stash/stash/scm/aeas/testci.git', '.']'> rc=128
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] releaseLocks(Git(repourl=Property(repository), workdir=Property(repopath))): []
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-]  step 'git' complete: failure (['update (failure)'])
deeptracy-server_1    | 172.18.0.4 - - [20/Sep/2018 17:56:43] "PUT /analysis/94ead550-81a8-467d-b771-90093c469a4f/extraction/failed HTTP/1.1" 200 0
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] releaseLocks(BackendSignal(method='PUT', name='Dependency Extraction Failed Signal', url=Interpolate('http://deeptracy-server:8088/analysis/%(prop:analysis_id)s/extraction/failed'), doStepIf=<function has_property.<locals>._has_property at 0x7f5431c4a598>, alwaysRun=True)): []
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-]  step 'Dependency Extraction Failed Signal' complete: success (None)
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-]  <Build bootstrap number:2 results:failure>: build finished
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] releaseLocks(<WasherDockerLatentWorker 'docker-0'>): []
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] disconnecting old worker docker-0 now
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] waiting for worker to finish disconnecting
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Worker.detached(docker-0)
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] releaseLocks(<WasherDockerLatentWorker 'docker-0'>): []
deeptracy-buildbot_1  | 2018-09-20 17:56:43+0000 [-] Stopping container 3d4c6e...
deeptracy-buildbot_1  | 2018-09-20 17:56:44+0000 [-] while insubstantiating
deeptracy-buildbot_1  |         Traceback (most recent call last):
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1475, in gotResult
deeptracy-buildbot_1  |             _inlineCallbacks(r, g, status)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
deeptracy-buildbot_1  |             result = result.throwExceptionIntoGenerator(g)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
deeptracy-buildbot_1  |             return g.throw(self.type, self.value, self.tb)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/worker/latent.py", line 253, in insubstantiate
deeptracy-buildbot_1  |             log.err(e, "while insubstantiating")
deeptracy-buildbot_1  |         --- <exception caught here> ---
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/worker/latent.py", line 248, in insubstantiate
deeptracy-buildbot_1  |             yield d
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
deeptracy-buildbot_1  |             result = inContext.theWork()
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
deeptracy-buildbot_1  |             inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
deeptracy-buildbot_1  |             return self.currentContext().callWithContext(ctx, func, *args, **kw)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
deeptracy-buildbot_1  |             return func(*args,**kw)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/worker/docker.py", line 307, in _thd_stop_instance
deeptracy-buildbot_1  |             docker_client.stop(instance['Id'])
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
deeptracy-buildbot_1  |             return f(self, resource_id, *args, **kwargs)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/docker/api/container.py", line 1120, in stop
deeptracy-buildbot_1  |             self._raise_for_status(res)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 231, in _raise_for_status
deeptracy-buildbot_1  |             raise create_api_error_from_http_exception(e)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
deeptracy-buildbot_1  |             raise cls(e, response=response, explanation=explanation)
deeptracy-buildbot_1  |         docker.errors.APIError: 500 Server Error: Internal Server Error ("cannot stop container: 241fad59d453476c28a17ea8bcd8834d289aadcfdb94073437cbc82c25e9da93: Cannot kill container 241fad59d453476c28a17ea8bcd8834d289aadcfdb94073437cbc82c25e9da93: unknown error after kill: docker-runc did not terminate sucessfully: container_linux.go:393: signaling init process caused "permission denied"
deeptracy-buildbot_1  |         : unknown")
deeptracy-buildbot_1  | 
deeptracy-buildbot_1  | 2018-09-20 17:56:44+0000 [Broker,6,172.18.0.1] worker 'docker-8' attaching from IPv4Address(type='TCP', host='172.18.0.1', port=42912)
deeptracy-buildbot_1  | 2018-09-20 17:56:44+0000 [Broker,6,172.18.0.1] Got workerinfo from 'docker-8'
deeptracy-buildbot_1  | 2018-09-20 17:56:44+0000 [Broker,6,172.18.0.1] Worker docker-8 received connection while not trying to substantiate.  Disconnecting.
deeptracy-buildbot_1  | 2018-09-20 17:56:44+0000 [Broker,6,172.18.0.1] waiting for worker to finish disconnecting
deeptracy-buildbot_1  | 2018-09-20 17:56:44+0000 [Broker,6,172.18.0.1] Peer will receive following PB traceback:
deeptracy-buildbot_1  | 2018-09-20 17:56:44+0000 [Broker,6,172.18.0.1] Unhandled Error
deeptracy-buildbot_1  |         Traceback (most recent call last):
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
deeptracy-buildbot_1  |             result = g.send(result)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/worker/protocols/pb.py", line 147, in attached
deeptracy-buildbot_1  |             yield self.worker.attached(self)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
deeptracy-buildbot_1  |             return _cancellableInlineCallbacks(gen)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
deeptracy-buildbot_1  |             _inlineCallbacks(None, g, status)
deeptracy-buildbot_1  |         --- <exception caught here> ---
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/worker/protocols/pb.py", line 147, in attached
deeptracy-buildbot_1  |             yield self.worker.attached(self)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
deeptracy-buildbot_1  |             result = g.send(result)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/worker/latent.py", line 153, in attached
deeptracy-buildbot_1  |             raise RuntimeError(msg)
deeptracy-buildbot_1  |         builtins.RuntimeError: Worker docker-8 received connection while not trying to substantiate.  Disconnecting.
deeptracy-buildbot_1  | 
deeptracy-buildbot_1  | 2018-09-20 17:56:45+0000 [-] while insubstantiating
deeptracy-buildbot_1  |         Traceback (most recent call last):
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1475, in gotResult
deeptracy-buildbot_1  |             _inlineCallbacks(r, g, status)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
deeptracy-buildbot_1  |             result = result.throwExceptionIntoGenerator(g)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
deeptracy-buildbot_1  |             return g.throw(self.type, self.value, self.tb)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/worker/latent.py", line 253, in insubstantiate
deeptracy-buildbot_1  |             log.err(e, "while insubstantiating")
deeptracy-buildbot_1  |         --- <exception caught here> ---
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/worker/latent.py", line 248, in insubstantiate
deeptracy-buildbot_1  |             yield d
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
deeptracy-buildbot_1  |             result = inContext.theWork()
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
deeptracy-buildbot_1  |             inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
deeptracy-buildbot_1  |             return self.currentContext().callWithContext(ctx, func, *args, **kw)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
deeptracy-buildbot_1  |             return func(*args,**kw)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/worker/docker.py", line 307, in _thd_stop_instance
deeptracy-buildbot_1  |             docker_client.stop(instance['Id'])
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
deeptracy-buildbot_1  |             return f(self, resource_id, *args, **kwargs)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/docker/api/container.py", line 1120, in stop
deeptracy-buildbot_1  |             self._raise_for_status(res)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 231, in _raise_for_status
deeptracy-buildbot_1  |             raise create_api_error_from_http_exception(e)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
deeptracy-buildbot_1  |             raise cls(e, response=response, explanation=explanation)
deeptracy-buildbot_1  |         docker.errors.APIError: 500 Server Error: Internal Server Error ("cannot stop container: 3d4c6e3b402125c837cd8bc2be29d33a43643931dcd027d96c468e9f6cbd2666: Cannot kill container 3d4c6e3b402125c837cd8bc2be29d33a43643931dcd027d96c468e9f6cbd2666: unknown error after kill: docker-runc did not terminate sucessfully: container_linux.go:393: signaling init process caused "permission denied"
deeptracy-buildbot_1  |         : unknown")
deeptracy-buildbot_1  | 
deeptracy-buildbot_1  | 2018-09-20 17:56:46+0000 [Broker,7,172.18.0.1] worker 'docker-0' attaching from IPv4Address(type='TCP', host='172.18.0.1', port=42916)
deeptracy-buildbot_1  | 2018-09-20 17:56:46+0000 [Broker,7,172.18.0.1] Got workerinfo from 'docker-0'
deeptracy-buildbot_1  | 2018-09-20 17:56:46+0000 [Broker,7,172.18.0.1] Worker docker-0 received connection while not trying to substantiate.  Disconnecting.
deeptracy-buildbot_1  | 2018-09-20 17:56:46+0000 [Broker,7,172.18.0.1] waiting for worker to finish disconnecting
deeptracy-buildbot_1  | 2018-09-20 17:56:46+0000 [Broker,7,172.18.0.1] Peer will receive following PB traceback:
deeptracy-buildbot_1  | 2018-09-20 17:56:46+0000 [Broker,7,172.18.0.1] Unhandled Error
deeptracy-buildbot_1  |         Traceback (most recent call last):
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
deeptracy-buildbot_1  |             result = g.send(result)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/worker/protocols/pb.py", line 147, in attached
deeptracy-buildbot_1  |             yield self.worker.attached(self)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
deeptracy-buildbot_1  |             return _cancellableInlineCallbacks(gen)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
deeptracy-buildbot_1  |             _inlineCallbacks(None, g, status)
deeptracy-buildbot_1  |         --- <exception caught here> ---
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/worker/protocols/pb.py", line 147, in attached
deeptracy-buildbot_1  |             yield self.worker.attached(self)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
deeptracy-buildbot_1  |             result = g.send(result)
deeptracy-buildbot_1  |           File "/usr/local/lib/python3.6/site-packages/buildbot/worker/latent.py", line 153, in attached
deeptracy-buildbot_1  |             raise RuntimeError(msg)
deeptracy-buildbot_1  |         builtins.RuntimeError: Worker docker-0 received connection while not trying to substantiate.  Disconnecting.
deeptracy-buildbot_1  | 

If I do a git clone outside the docker I'm able to do a clone without problems (no credentials are required):

berni@ubuntu-ci:~$ git clone http://stash/stash/scm/aeas/testci.git
Cloning into 'testci'...
remote: Counting objects: 33, done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 33 (delta 3), reused 0 (delta 0)
Unpacking objects: 100% (33/33), done.

Could you help me with this issue?

Thanks for your time

Get scans by project ID

Add an empoint to get all scan issued by a given project

Add false positive attribute to vulnerabilities

Ability to mark a vulnerability / library as a false positive will be desirable to avoid reporting things that we know are not true :D

Front dashboard

Make a front dashboard to be able to browse projects, scans and vulnerabilities.

• Public repository
• Docker image with the dashboard
• Documentation

Create a project directly from a webhook

We want to be able to create projects directly from a webhook (GitHub&Bitbucket).

This feature can be only achieved for PUBLIC projects.
This feature involves making a pull for the repository and determining the default language (by examining the files in the repo) for future scans.

FIX API Docs

API Docs generating crash on "make html"

Search by term on Server

Change PostgresSQL Driver: psycopg2 -> pg8000

Currently the Postgres driver for Alchemy is psycopg2 (LGPL). Their license is incompatible with deeptracy acceptance terms.

Change psycopg2 -> pg8000 (BSD)

Change needed in the connection string:

postgresql+psycopg2://user:password@host:port/dbname

To

postgresql+pg8000://user:password@host:port/dbname

error: version_part = pattern.split(parts[1])[1]

Good morning. Good morning.
First of all thank you very much for this project. It was absolutely necessary.
I'm having some trouble getting into deeptracy + patton.

First I followed patton's step-by-step instructions: https://patton-server.readthedocs.io/en/latest/quickstart.html#first-of-run-patton-server.
This one looks like it's raised well.

Then I tried to start the deeptracy service https://deeptracy.readthedocs.io/en/latest/installation.html#bringing-up-the-environment. When I create a project and a scan, the scan fails.

I did this requests:

curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"repo":"https://github.com/BBVA/deeptracy-api.git","name":"deeptracy-api-test"}' \
  http://localhost:80/api/1/project/

curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"project_id":"54ef977d835c48459ba85c2e9e82931f","lang":"python"}' \
  http://localhost:80/api/1/scan/

And the log output is....

deeptracy_1      | Traceback (most recent call last):
deeptracy_1      |   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 374, in trace_task
deeptracy_1      |     R = retval = fun(*args, **kwargs)
deeptracy_1      |   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 629, in __protected_call__
deeptracy_1      |     return self.run(*args, **kwargs)
deeptracy_1      |   File "/usr/local/lib/python3.6/site-packages/deeptracy/tasks/scan_deps.py", line 41, in scan_deps
deeptracy_1      |     scan_deps = get_dependencies(scan.lang, scan.source_path)
deeptracy_1      |   File "/usr/local/lib/python3.6/site-packages/deeptracy/tasks/scan_deps.py", line 88, in get_dependencies
deeptracy_1      |     return get_dependencies_for_python(sources, mounted_vol, docker_volumes)
deeptracy_1      |   File "/usr/local/lib/python3.6/site-packages/deeptracy/tasks/scan_deps.py", line 239, in get_dependencies_for_python
deeptracy_1      |     version_part = pattern.split(parts[1])[1]
deeptracy_1      | IndexError: list index out of range

I don't know what's going on. Could you help me???
Thanks!

Add support for multi-branches in scans

Right now all scans are made in the master branch, we need to support scan in other branches as well.

• Get branch from webhooks from github/bitbucket
• Get branch in POST /scan API endpoint
• Save branch in scan entity and use it when cloning the repository

Create deploy template for openshift

Provide a openshift template to deploy the complete deeptracy solution

related #30

Add development builds releases to deeptracy-core

When on branch develop or feature branches deeptracy-core needs to release to pypi with development builds.

• Change bumpversion regexp and patterns
• Add job to travis when on develop or feature branches

Track scan states

Right now the scans are always in PENDING state. We need to track its state for the whole process and change it according to the process.

change defaults env

change defaults env examples

Configurations in database

We need an entity to store configurations in the database.

ALLOWED_SCANS_PER_PERIOD and ALLOWED_SCANS_CHECK_PERIOD should be stored in the database and should be editable values.

Add project name

We need a project name property in project entities

Get scan vulnerabilities

Get a list of vulnerabilities found for a given scan

Suggestion for capturing more deps

Hi there!
Excellent tool!
I am the maintainer of https://github.com/nexB/scancode-toolkit .... and it does parse a few package manifests including capturing direct (or full locked) deps. You may want to check it out, this is in Python.

Check also https://github.com/heremaps/oss-review-toolkit that has a similar approach to yours to capture deps using package managers as you do, but is in kotlin.

Email notifications

Add support to notify by emails the vulnerabilities on each scan

Handle errors inside celery tasks

We need to handle errors (exceptions) inside celery task, and act according (updating scan state for example)

This is related to #39 as we need a way to be able to tests this behaviours.

http://docs.celeryproject.org/en/latest/userguide/canvas.html#error-handling

Do not include tests package in wheel

When building the wheel to push to pypi, avoid adding the tests package in the bundle.

Add default_language to project

Project should have a default language to process scans.

If a scan is launched for a project without specifying the language de default language for the project must be used.

This issue comes from the need to launch scans directly from webhooks (bitbucket or github) and not having the language available.

Add support for github push action webhooks

Add support for github webhooks for pushes actions.

When a user adds a webhook from github to deeptracy, we need to handle their pushes to launch scans for their projects.

Generate and save dependency list for nodejs projects

When we scan a nodejs project we need to generate and save in the database a dependency list.

This list is needed to dynamic find new vulnerabilities when they are found and published without the needing of scanning again projects.

This issue is the first step to achieve on of the main features of deeptracy, that is find vulnerabilities when they are found in libraries without the need of scaning your project

Add extra info in notifications

When sending slack notifications, we should include more info, number of vulnerabilities, a list with them...

Make error - docker

After make start, following error appear

Pulling deeptracy-buildbot (deeptracy-buildbot:)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

Add support to open github issues

When a scan is finished, we want to open a issue directly in GitHub with the vulnerability list.

To be able to get this feature we need more info at project creation time, mostly the credentials to be able to open the issues.

Merge plugin results

Merge task is not impemented

Link deeptracy-core with pip in development mode

With pip we can link the deeptracy-core requirements to the projects instead of the installed wheel. This way the development flow is more easy.

Fix docker badges in readme

Docker badges in readme are not showing properly. We should use the build badge instead of the automaged one, but i can`t get it working.

The badge can be picked from: https://shields.io/

Patton

Add patton to deeptracy

• Make public repository for patton
• Integrate patton with deeptracy
• Patton docker images
• Documentation