bbva / deeptracy Goto Github PK
View Code? Open in Web Editor NEWThe Security Dependency Orchestrator Service
The Security Dependency Orchestrator Service
We need an entity to store configurations in the database.
ALLOWED_SCANS_PER_PERIOD and ALLOWED_SCANS_CHECK_PERIOD should be stored in the database and should be editable values.
Make a front dashboard to be able to browse projects, scans and vulnerabilities.
Hi everyone,
I've just followed the installation docs of the new version of deeptracy (on a fresh install of ubuntu server 18.04), I prepared a demo repo with this configuration (.deeptracy.yml):
projects:
TestCi:
type: deeptracy-mvn:3.5-jdk-8
strategy: mvn_dependencytree
unimportant: false
config:
path: src/main
But when I ran this call:
curl --data '{"repository": "http://stash/stash/scm/aeas/testci.git", "commit": "17e74a2a72cd15539f04d1ab888a4a98c13a1b65"}' -XPOST http://localhost:8088/analysis/ -H "Content-Type: application/json"
I saw in the logs this error:
deeptracy-server_1 | 172.18.0.1 - - [20/Sep/2018 17:56:39] "POST /analysis/ HTTP/1.1" 200 46
deeptracy-worker_1 | [2018-09-20 17:56:39,022: INFO/MainProcess] Received task: deeptracy.tasks.request_extraction[78e1b7b4-81d6-4e96-b356-54c2450226e1]
deeptracy-buildbot_1 | 2018-09-20 17:56:39+0000 [_GenericHTTPChannelProtocol,1,172.18.0.5] WARNING: change source is using deprecated self.master.addChange method; this method will disappear in Buildbot-1.0.0
deeptracy-buildbot_1 | 2018-09-20 17:56:39+0000 [-] added change with revision 17e74a2a72cd15539f04d1ab888a4a98c13a1b65 to database
deeptracy-buildbot_1 | 2018-09-20 17:56:39+0000 [-] injected change Change(revision='17e74a2a72cd15539f04d1ab888a4a98c13a1b65', who='deeptracy', branch=None, comments='', when=1537466199, category=None, project='http://stash/stash/scm/aeas/testci.git', repository='http://stash/stash/scm/aeas/testci.git', codebase='')
deeptracy-worker_1 | /usr/local/lib/python3.6/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is
deeptracy-worker_1 | absolutely not recommended!
deeptracy-worker_1 |
deeptracy-worker_1 | Please specify a different user using the --uid option.
deeptracy-worker_1 |
deeptracy-worker_1 | User information: uid=0 euid=0 gid=0 egid=0
deeptracy-worker_1 |
deeptracy-worker_1 | uid=uid, euid=euid, gid=gid, egid=egid,
deeptracy-worker_1 | [2018-09-20 17:56:39,159: INFO/ForkPoolWorker-3] Task deeptracy.tasks.request_extraction[78e1b7b4-81d6-4e96-b356-54c2450226e1] succeeded in 0.1342389319997892s: None
deeptracy-buildbot_1 | 2018-09-20 17:56:40+0000 [-] added buildset 3 to database
deeptracy-buildbot_1 | 2018-09-20 17:56:40+0000 [-] starting build <Build launch number:None results:success> using worker <LatentWorkerForBuilder builder='launch' worker='docker-8' state=AVAILABLE>
deeptracy-buildbot_1 | 2018-09-20 17:56:40+0000 [-] <Build launch number:None results:success>.startBuild
deeptracy-buildbot_1 | 2018-09-20 17:56:40+0000 [-] acquireLocks(worker <WasherDockerLatentWorker 'docker-8'>, locks [])
deeptracy-buildbot_1 | 2018-09-20 17:56:40+0000 [-] substantiating worker <LatentWorkerForBuilder builder='launch' worker='docker-8' state=DETACHED>
deeptracy-buildbot_1 | 2018-09-20 17:56:40+0000 [-] Container created, Id: 241fad...
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [-] Container started
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [-] docker VM 241fad: b'2018-09-20T17:56:41+0000 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 18.4.0 (/usr/local/bin/python 3.6.5) starting up.\n2018-09-20T17:56:41+0000 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.'
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [-] docker VM 241fad: b'2018-09-20T17:56:41+0000 [-] Starting Worker -- version: 1970.01.01'
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [-] docker VM 241fad: b'2018-09-20T17:56:41+0000 [-] recording hostname in twistd.hostname'
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [-] docker VM 241fad: b'2018-09-20T17:56:41+0000 [buildbot_worker.pb.BotFactory#info] Starting factory <buildbot_worker.pb.BotFactory object at 0x7f7c63e70b38>\n2018-09-20T17:56:41+0000 [-] Connecting to 172.17.0.1:9989'
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] worker 'docker-8' attaching from IPv4Address(type='TCP', host='172.18.0.1', port=42900)
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [-] docker VM 241fad: b'2018-09-20T17:56:41+0000 [HangCheckProtocol,client] message from master: attached'
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] Got workerinfo from 'docker-8'
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [-] bot attached
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [-] docker VM 241fad: b"2018-09-20T17:56:41+0000 [HangCheckProtocol,client] I have a leftover directory 'zope' that is not being used by the buildmaster: you can delete it now"
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] Worker docker-8 attached to launch
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] Worker docker-8 attached to analyze
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] Worker docker-8 attached to bootstrap
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] Worker docker-8 substantiated \o/
deeptracy-buildbot_1 | 2018-09-20 17:56:41+0000 [Broker,4,172.18.0.1] Firing docker-8 substantiation deferred with success
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] starting build <Build launch number:2 results:success>.. pinging the worker <LatentWorkerForBuilder builder='launch' worker='docker-8' state=BUILDING>
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] sending ping
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [Broker,4,172.18.0.1] ping finished: success
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] added buildset 4 to database
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] releaseLocks(Trigger(schedulerNames=['bootstrap'], set_properties={'virtual_builder_name': Property(project)})): []
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] step 'trigger' complete: success (None)
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] <Build launch number:2 results:success>: build finished
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] starting build <Build bootstrap number:None results:success> using worker <LatentWorkerForBuilder builder='bootstrap' worker='docker-0' state=AVAILABLE>
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] <Build bootstrap number:None results:success>.startBuild
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] releaseLocks(<WasherDockerLatentWorker 'docker-8'>): []
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] disconnecting old worker docker-8 now
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] waiting for worker to finish disconnecting
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [Broker,4,172.18.0.1] Worker.detached(docker-8)
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] Stopping container 241fad...
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [Broker,4,172.18.0.1] releaseLocks(<WasherDockerLatentWorker 'docker-8'>): []
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] acquireLocks(worker <WasherDockerLatentWorker 'docker-0'>, locks [])
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] substantiating worker <LatentWorkerForBuilder builder='bootstrap' worker='docker-0' state=DETACHED>
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] Container created, Id: 3d4c6e...
deeptracy-buildbot_1 | 2018-09-20 17:56:42+0000 [-] Container started
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] docker VM 3d4c6e: b'2018-09-20T17:56:43+0000 [twisted.scripts._twistd_unix.UnixAppLogger#info] twistd 18.4.0 (/usr/local/bin/python 3.6.5) starting up.'
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] docker VM 3d4c6e: b'2018-09-20T17:56:43+0000 [twisted.scripts._twistd_unix.UnixAppLogger#info] reactor class: twisted.internet.epollreactor.EPollReactor.\n2018-09-20T17:56:43+0000 [-] Starting Worker -- version: 1970.01.01\n2018-09-20T17:56:43+0000 [-] recording hostname in twistd.hostname'
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] docker VM 3d4c6e: b'2018-09-20T17:56:43+0000 [buildbot_worker.pb.BotFactory#info] Starting factory <buildbot_worker.pb.BotFactory object at 0x7feac505dba8>'
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] docker VM 3d4c6e: b'2018-09-20T17:56:43+0000 [-] Connecting to 172.17.0.1:9989'
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] worker 'docker-0' attaching from IPv4Address(type='TCP', host='172.18.0.1', port=42904)
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] docker VM 3d4c6e: b'2018-09-20T17:56:43+0000 [HangCheckProtocol,client] message from master: attached'
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Got workerinfo from 'docker-0'
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] bot attached
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] docker VM 3d4c6e: b"2018-09-20T17:56:43+0000 [HangCheckProtocol,client] I have a leftover directory 'zope' that is not being used by the buildmaster: you can delete it now\n2018-09-20T17:56:43+0000 [HangCheckProtocol,client] I have a leftover directory 'sqlalchemy' that is not being used by the buildmaster: you can delete it now"
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Worker docker-0 attached to launch
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Worker docker-0 attached to analyze
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Worker docker-0 attached to bootstrap
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Worker docker-0 substantiated \o/
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Firing docker-0 substantiation deferred with success
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] starting build <Build bootstrap number:2 results:success>.. pinging the worker <LatentWorkerForBuilder builder='bootstrap' worker='docker-0' state=BUILDING>
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] sending ping
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] ping finished: success
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] releaseLocks(SetProperty('repopath', '/work')): []
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] step 'SetProperty' complete: success (None)
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] <RemoteShellCommand '['git', '--version']'>: RemoteCommand.run [4]
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] command '['git', '--version']' in dir '/work'
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] <RemoteShellCommand '['git', '--version']'> rc=0
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] <RemoteCommand 'stat' at 139999432269776>: RemoteCommand.run [5]
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] <RemoteCommand 'stat' at 139999432269776> rc=2
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] <RemoteCommand 'listdir' at 139999559733936>: RemoteCommand.run [6]
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] <RemoteCommand 'listdir' at 139999559733936> rc=0
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] No git repo present, making full clone
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] <RemoteShellCommand '['git', 'clone', 'http://stash/stash/scm/aeas/testci.git', '.']'>: RemoteCommand.run [7]
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] command '['git', 'clone', 'http://stash/stash/scm/aeas/testci.git', '.']' in dir '/work'
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] <RemoteShellCommand '['git', 'clone', 'http://stash/stash/scm/aeas/testci.git', '.']'> rc=128
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] releaseLocks(Git(repourl=Property(repository), workdir=Property(repopath))): []
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] step 'git' complete: failure (['update (failure)'])
deeptracy-server_1 | 172.18.0.4 - - [20/Sep/2018 17:56:43] "PUT /analysis/94ead550-81a8-467d-b771-90093c469a4f/extraction/failed HTTP/1.1" 200 0
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] releaseLocks(BackendSignal(method='PUT', name='Dependency Extraction Failed Signal', url=Interpolate('http://deeptracy-server:8088/analysis/%(prop:analysis_id)s/extraction/failed'), doStepIf=<function has_property.<locals>._has_property at 0x7f5431c4a598>, alwaysRun=True)): []
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] step 'Dependency Extraction Failed Signal' complete: success (None)
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] <Build bootstrap number:2 results:failure>: build finished
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] releaseLocks(<WasherDockerLatentWorker 'docker-0'>): []
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] disconnecting old worker docker-0 now
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] waiting for worker to finish disconnecting
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] Worker.detached(docker-0)
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [Broker,5,172.18.0.1] releaseLocks(<WasherDockerLatentWorker 'docker-0'>): []
deeptracy-buildbot_1 | 2018-09-20 17:56:43+0000 [-] Stopping container 3d4c6e...
deeptracy-buildbot_1 | 2018-09-20 17:56:44+0000 [-] while insubstantiating
deeptracy-buildbot_1 | Traceback (most recent call last):
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1475, in gotResult
deeptracy-buildbot_1 | _inlineCallbacks(r, g, status)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
deeptracy-buildbot_1 | result = result.throwExceptionIntoGenerator(g)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
deeptracy-buildbot_1 | return g.throw(self.type, self.value, self.tb)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/worker/latent.py", line 253, in insubstantiate
deeptracy-buildbot_1 | log.err(e, "while insubstantiating")
deeptracy-buildbot_1 | --- <exception caught here> ---
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/worker/latent.py", line 248, in insubstantiate
deeptracy-buildbot_1 | yield d
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
deeptracy-buildbot_1 | result = inContext.theWork()
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
deeptracy-buildbot_1 | inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
deeptracy-buildbot_1 | return self.currentContext().callWithContext(ctx, func, *args, **kw)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
deeptracy-buildbot_1 | return func(*args,**kw)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/worker/docker.py", line 307, in _thd_stop_instance
deeptracy-buildbot_1 | docker_client.stop(instance['Id'])
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
deeptracy-buildbot_1 | return f(self, resource_id, *args, **kwargs)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/docker/api/container.py", line 1120, in stop
deeptracy-buildbot_1 | self._raise_for_status(res)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 231, in _raise_for_status
deeptracy-buildbot_1 | raise create_api_error_from_http_exception(e)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
deeptracy-buildbot_1 | raise cls(e, response=response, explanation=explanation)
deeptracy-buildbot_1 | docker.errors.APIError: 500 Server Error: Internal Server Error ("cannot stop container: 241fad59d453476c28a17ea8bcd8834d289aadcfdb94073437cbc82c25e9da93: Cannot kill container 241fad59d453476c28a17ea8bcd8834d289aadcfdb94073437cbc82c25e9da93: unknown error after kill: docker-runc did not terminate sucessfully: container_linux.go:393: signaling init process caused "permission denied"
deeptracy-buildbot_1 | : unknown")
deeptracy-buildbot_1 |
deeptracy-buildbot_1 | 2018-09-20 17:56:44+0000 [Broker,6,172.18.0.1] worker 'docker-8' attaching from IPv4Address(type='TCP', host='172.18.0.1', port=42912)
deeptracy-buildbot_1 | 2018-09-20 17:56:44+0000 [Broker,6,172.18.0.1] Got workerinfo from 'docker-8'
deeptracy-buildbot_1 | 2018-09-20 17:56:44+0000 [Broker,6,172.18.0.1] Worker docker-8 received connection while not trying to substantiate. Disconnecting.
deeptracy-buildbot_1 | 2018-09-20 17:56:44+0000 [Broker,6,172.18.0.1] waiting for worker to finish disconnecting
deeptracy-buildbot_1 | 2018-09-20 17:56:44+0000 [Broker,6,172.18.0.1] Peer will receive following PB traceback:
deeptracy-buildbot_1 | 2018-09-20 17:56:44+0000 [Broker,6,172.18.0.1] Unhandled Error
deeptracy-buildbot_1 | Traceback (most recent call last):
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
deeptracy-buildbot_1 | result = g.send(result)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/worker/protocols/pb.py", line 147, in attached
deeptracy-buildbot_1 | yield self.worker.attached(self)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
deeptracy-buildbot_1 | return _cancellableInlineCallbacks(gen)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
deeptracy-buildbot_1 | _inlineCallbacks(None, g, status)
deeptracy-buildbot_1 | --- <exception caught here> ---
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/worker/protocols/pb.py", line 147, in attached
deeptracy-buildbot_1 | yield self.worker.attached(self)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
deeptracy-buildbot_1 | result = g.send(result)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/worker/latent.py", line 153, in attached
deeptracy-buildbot_1 | raise RuntimeError(msg)
deeptracy-buildbot_1 | builtins.RuntimeError: Worker docker-8 received connection while not trying to substantiate. Disconnecting.
deeptracy-buildbot_1 |
deeptracy-buildbot_1 | 2018-09-20 17:56:45+0000 [-] while insubstantiating
deeptracy-buildbot_1 | Traceback (most recent call last):
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1475, in gotResult
deeptracy-buildbot_1 | _inlineCallbacks(r, g, status)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
deeptracy-buildbot_1 | result = result.throwExceptionIntoGenerator(g)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
deeptracy-buildbot_1 | return g.throw(self.type, self.value, self.tb)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/worker/latent.py", line 253, in insubstantiate
deeptracy-buildbot_1 | log.err(e, "while insubstantiating")
deeptracy-buildbot_1 | --- <exception caught here> ---
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/worker/latent.py", line 248, in insubstantiate
deeptracy-buildbot_1 | yield d
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
deeptracy-buildbot_1 | result = inContext.theWork()
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
deeptracy-buildbot_1 | inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
deeptracy-buildbot_1 | return self.currentContext().callWithContext(ctx, func, *args, **kw)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
deeptracy-buildbot_1 | return func(*args,**kw)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/worker/docker.py", line 307, in _thd_stop_instance
deeptracy-buildbot_1 | docker_client.stop(instance['Id'])
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
deeptracy-buildbot_1 | return f(self, resource_id, *args, **kwargs)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/docker/api/container.py", line 1120, in stop
deeptracy-buildbot_1 | self._raise_for_status(res)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/docker/api/client.py", line 231, in _raise_for_status
deeptracy-buildbot_1 | raise create_api_error_from_http_exception(e)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
deeptracy-buildbot_1 | raise cls(e, response=response, explanation=explanation)
deeptracy-buildbot_1 | docker.errors.APIError: 500 Server Error: Internal Server Error ("cannot stop container: 3d4c6e3b402125c837cd8bc2be29d33a43643931dcd027d96c468e9f6cbd2666: Cannot kill container 3d4c6e3b402125c837cd8bc2be29d33a43643931dcd027d96c468e9f6cbd2666: unknown error after kill: docker-runc did not terminate sucessfully: container_linux.go:393: signaling init process caused "permission denied"
deeptracy-buildbot_1 | : unknown")
deeptracy-buildbot_1 |
deeptracy-buildbot_1 | 2018-09-20 17:56:46+0000 [Broker,7,172.18.0.1] worker 'docker-0' attaching from IPv4Address(type='TCP', host='172.18.0.1', port=42916)
deeptracy-buildbot_1 | 2018-09-20 17:56:46+0000 [Broker,7,172.18.0.1] Got workerinfo from 'docker-0'
deeptracy-buildbot_1 | 2018-09-20 17:56:46+0000 [Broker,7,172.18.0.1] Worker docker-0 received connection while not trying to substantiate. Disconnecting.
deeptracy-buildbot_1 | 2018-09-20 17:56:46+0000 [Broker,7,172.18.0.1] waiting for worker to finish disconnecting
deeptracy-buildbot_1 | 2018-09-20 17:56:46+0000 [Broker,7,172.18.0.1] Peer will receive following PB traceback:
deeptracy-buildbot_1 | 2018-09-20 17:56:46+0000 [Broker,7,172.18.0.1] Unhandled Error
deeptracy-buildbot_1 | Traceback (most recent call last):
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
deeptracy-buildbot_1 | result = g.send(result)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/worker/protocols/pb.py", line 147, in attached
deeptracy-buildbot_1 | yield self.worker.attached(self)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
deeptracy-buildbot_1 | return _cancellableInlineCallbacks(gen)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
deeptracy-buildbot_1 | _inlineCallbacks(None, g, status)
deeptracy-buildbot_1 | --- <exception caught here> ---
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/worker/protocols/pb.py", line 147, in attached
deeptracy-buildbot_1 | yield self.worker.attached(self)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
deeptracy-buildbot_1 | result = g.send(result)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/worker/latent.py", line 153, in attached
deeptracy-buildbot_1 | raise RuntimeError(msg)
deeptracy-buildbot_1 | builtins.RuntimeError: Worker docker-0 received connection while not trying to substantiate. Disconnecting.
deeptracy-buildbot_1 |
If I do a git clone outside the docker I'm able to do a clone without problems (no credentials are required):
berni@ubuntu-ci:~$ git clone http://stash/stash/scm/aeas/testci.git
Cloning into 'testci'...
remote: Counting objects: 33, done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 33 (delta 3), reused 0 (delta 0)
Unpacking objects: 100% (33/33), done.
Could you help me with this issue?
Thanks for your time
We want to be able to create projects directly from a webhook (GitHub&Bitbucket).
This feature can be only achieved for PUBLIC projects.
This feature involves making a pull for the repository and determining the default language (by examining the files in the repo) for future scans.
Docker badges in readme are not showing properly. We should use the build
badge instead of the automaged
one, but i can`t get it working.
The badge can be picked from: https://shields.io/
The master output this warning when a browser connects to it:
deeptracy-buildbot_1 | Traceback (most recent call last):
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/twisted/web/server.py", line 481, in getSession
deeptracy-buildbot_1 | session = self.site.getSession(sessionCookie)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/www/service.py", line 174, in getSession
deeptracy-buildbot_1 | return BuildbotSession(self, uid)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/www/service.py", line 74, in __init__
deeptracy-buildbot_1 | self._fromToken(token)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/www/service.py", line 88, in _fromToken
deeptracy-buildbot_1 | log.err(e, "while decoding JWT session")
deeptracy-buildbot_1 | --- <exception caught here> ---
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/buildbot/www/service.py", line 84, in _fromToken
deeptracy-buildbot_1 | SESSION_SECRET_ALGORITHM])
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/jwt/api_jwt.py", line 93, in decode
deeptracy-buildbot_1 | jwt, key=key, algorithms=algorithms, options=options, **kwargs
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/jwt/api_jws.py", line 157, in decode
deeptracy-buildbot_1 | key, algorithms)
deeptracy-buildbot_1 | File "/usr/local/lib/python3.6/site-packages/jwt/api_jws.py", line 224, in _verify_signature
deeptracy-buildbot_1 | raise InvalidSignatureError('Signature verification failed')
deeptracy-buildbot_1 | jwt.exceptions.InvalidSignatureError: Signature verification failed
deeptracy-buildbot_1 |
Related to #96 .
The two images we are building in docker hub needs info, description, links and such.
change defaults env examples
When a scan is finished, we want to open a issue directly in Jira with the vulnerability list.
To be able to get this feature we need more info at project creation time, mostly the credentials to be able to open the issues.
Realted #28
Provide a rancher template to deploy the complete deeptracy solution
When upload develop changes in api or workers, if this changes depends on deeptracy-core, get the master packages on pip repository, and fails. The task is generate a dev pip packages for pipelines, to avoid this fails.
Add support to notify by emails the vulnerabilities on each scan
Scan need to have the following properties:
total_packages
total_vulnerabilties
Merge task is not impemented
Now we are storing intermediate state like scan.analysis_count
and scan.analysis_done
.
If celery brokes, or a plugin fails this should reflect the incomplete state.
Now it's difficult because locally the environment is manually loaded before running behave.
We should normalize local and travis behaves to ease that and add this test.
After make start, following error appear
Pulling deeptracy-buildbot (deeptracy-buildbot:)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.
Currently the Postgres driver for Alchemy is psycopg2 (LGPL). Their license is incompatible with deeptracy acceptance terms.
Change psycopg2 -> pg8000 (BSD)
Change needed in the connection string:
postgresql+psycopg2://user:password@host:port/dbname
To
postgresql+pg8000://user:password@host:port/dbname
Ability to mark a vulnerability / library as a false positive will be desirable to avoid reporting things that we know are not true :D
When sending slack notifications, we should include more info, number of vulnerabilities, a list with them...
Add an empoint to get all scan issued by a given project
Write a complete user documentation and publish it to read the docs.
This documentation should include API, usage and deployment documentation
Provide a openshift template to deploy the complete deeptracy solution
related #30
We need a project name property in project entities
Hi there!
Excellent tool!
I am the maintainer of https://github.com/nexB/scancode-toolkit .... and it does parse a few package manifests including capturing direct (or full locked) deps. You may want to check it out, this is in Python.
Check also https://github.com/heremaps/oss-review-toolkit that has a similar approach to yours to capture deps using package managers as you do, but is in kotlin.
Now behave is configured to launch docker-compose (test/acceptance/environment.py
) after launching the tests. This dependency should be removed. The environment should be provided by externals sources (maybe in the make file or in travis).
With pip we can link the deeptracy-core requirements to the projects instead of the installed wheel. This way the development flow is more easy.
Get a list of vulnerabilities found for a given scan
Add patton to deeptracy
With visibility to our code coverage we are going to be more encouraged to rise that coverage (right now is poor) on every pull request.
Hi, trhanks for distribute this software seems pretty interesting, I found that in:
https://www.bbva.com/es/gestion-vulnerabilidades-dependencias-entornos-cicd-herramientas-open-
source/
I've configured a my env with the following docker-compose:
version: '3'
services:
postgres:
image: postgres:9.6-alpine
environment:
- POSTGRES_PASSWORD=postgres
ports:
- 5433:5433
command: -p 5433
redis:
image: redis:3-alpine
ports:
- 6380:6380
deeptracy:
image: bbvalabs/deeptracy
depends_on:
- redis
- postgres
environment:
- BROKER_URI=redis://redis:6379
- DATABASE_URI=postgresql://postgres:postgres@postgres:5433/deeptracy
- POSTGRES_URI=postgresql://postgres:postgres@postgres:5433
- SHARED_VOLUME_PATH=/tmp/deeptracy
- LOCAL_PRIVATE_KEY_FILE=/root/.ssh/id_rsa
- PATTON_URI=http://0.0.0.0:8000
# - EMAIL_SMTP_SERVER=xxx.xxx.xxx
# - EMAIL_SMTP_PORT=xxx
# - [email protected]
# - EMAIL_SMTP_PASSWORD=xxxxx
# - [email protected]
ports:
- 8000:8000
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /tmp:/tmp
- ./private_key:/root/.ssh/
privileged: true
command: ["./init_patton_db.sh"]
patton-server:
image: bbvalabs/patton-server
environment:
- WORKERS=1
- BACKLOG=512
- LISTEN_PORT=9000
- POSTGRES_HOST=postgres
- POSTGRES_PORT=5433
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=patton
ports:
- 9000:8000
depends_on:
- postgres
deeptracy-api:
image: bbvalabs/deeptracy-api
depends_on:
- redis
- postgres
- deeptracy
ports:
- 8081:8081
environment:
- BROKER_URI=redis://redis:6380
- DATABASE_URI=postgresql://postgres:postgres@postgres:5433/deeptracy
- SERVER_ADDRESS=0.0.0.0:8081
- GUNICORN_WORKERS=1
- LOG_LEVEL=INFO
command: ["./wait-for-it.sh", "postgres:5433", "--", "/opt/deeptracy/run.sh"]
Also I've configured a webhook on my git repository (stash), I've configured as follows:
But when the hook is launched I'm seeing a "BAD REQUEST" in deeptracy-api logs without extra information:
deeptracy_1 | [2018-09-04 12:47:01,432: INFO/MainProcess] Connected to redis://redis:6379//
deeptracy_1 | [2018-09-04 12:47:01,453: INFO/MainProcess] mingle: searching for neighbors
deeptracy_1 | [2018-09-04 12:47:02,496: INFO/MainProcess] mingle: all alone
deeptracy_1 | [2018-09-04 12:47:02,520: INFO/MainProcess] celery@fb67cffc785d ready.
> nvdcve-2.0-2003.xml.gz: 440kB [00:00, 575kB/s] 00<00:00, 331kB/s]
> nvdcve-2.0-2005.xml.gz: 1.35MB [00:01, 1.27MB/s] <00:00, 607kB/s]
> nvdcve-2.0-2015.xml.gz: 2.35MB [00:01, 1.74MB/s] 1<00:00, 1.01MB/s]
> nvdcve-2.0-2009.xml.gz: 2.15MB [00:01, 1.79MB/s] 01<00:00, 895kB/s]
> nvdcve-2.0-2011.xml.gz: 6.32MB [00:01, 3.68MB/s] :01<00:00, 2.61MB/s]
> nvdcve-2.0-2016.xml.gz: 2.75MB [00:03, 865kB/s] 0:03<00:00, 975kB/s]
> nvdcve-2.0-2008.xml.gz: 2.22MB [00:02, 836kB/s] 0:02<00:00, 960kB/s]
> nvdcve-2.0-2014.xml.gz: 2.79MB [00:01, 2.10MB/s] 1<00:00, 1.16MB/s]
> official-cpe-dictionary_v2.3.xml.gz: 2.57MB [00:01, 1.93MB/s] 1<00:00, 1.13MB/s]
> nvdcve-2.0-2010.xml.gz: 2.90MB [00:01, 1.30MB/s] <00:00, 960kB/s]
> nvdcve-2.0-2018.xml.gz: 2.89MB [00:01, 2.22MB/s] 1<00:00, 1.14MB/s]
> nvdcve-2.0-2006.xml.gz: 2.11MB [00:01, 1.96MB/s] <00:00, 774kB/s]
deeptracy-api_1 | [2018-09-04 12:55:20,371 deeptracy INFO ] 192.168.151.132 POST http /api/1/webhook/? 400 BAD REQUEST
Could you help me with that ?
When a scan is finished, we want to open a issue directly in GitHub with the vulnerability list.
To be able to get this feature we need more info at project creation time, mostly the credentials to be able to open the issues.
Good morning. Good morning.
First of all thank you very much for this project. It was absolutely necessary.
I'm having some trouble getting into deeptracy + patton.
First I followed patton's step-by-step instructions: https://patton-server.readthedocs.io/en/latest/quickstart.html#first-of-run-patton-server
.
This one looks like it's raised well.
Then I tried to start the deeptracy service https://deeptracy.readthedocs.io/en/latest/installation.html#bringing-up-the-environment
. When I create a project and a scan, the scan fails.
I did this requests:
curl --header "Content-Type: application/json" \
--request POST \
--data '{"repo":"https://github.com/BBVA/deeptracy-api.git","name":"deeptracy-api-test"}' \
http://localhost:80/api/1/project/
curl --header "Content-Type: application/json" \
--request POST \
--data '{"project_id":"54ef977d835c48459ba85c2e9e82931f","lang":"python"}' \
http://localhost:80/api/1/scan/
And the log output is....
deeptracy_1 | Traceback (most recent call last):
deeptracy_1 | File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 374, in trace_task
deeptracy_1 | R = retval = fun(*args, **kwargs)
deeptracy_1 | File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 629, in __protected_call__
deeptracy_1 | return self.run(*args, **kwargs)
deeptracy_1 | File "/usr/local/lib/python3.6/site-packages/deeptracy/tasks/scan_deps.py", line 41, in scan_deps
deeptracy_1 | scan_deps = get_dependencies(scan.lang, scan.source_path)
deeptracy_1 | File "/usr/local/lib/python3.6/site-packages/deeptracy/tasks/scan_deps.py", line 88, in get_dependencies
deeptracy_1 | return get_dependencies_for_python(sources, mounted_vol, docker_volumes)
deeptracy_1 | File "/usr/local/lib/python3.6/site-packages/deeptracy/tasks/scan_deps.py", line 239, in get_dependencies_for_python
deeptracy_1 | version_part = pattern.split(parts[1])[1]
deeptracy_1 | IndexError: list index out of range
I don't know what's going on. Could you help me???
Thanks!
When on branch develop or feature branches deeptracy-core needs to release to pypi with development builds.
Right now the scans are always in PENDING state. We need to track its state for the whole process and change it according to the process.
We need to handle errors (exceptions) inside celery task, and act according (updating scan state for example)
This is related to #39 as we need a way to be able to tests this behaviours.
http://docs.celeryproject.org/en/latest/userguide/canvas.html#error-handling
Add support for github webhooks for pushes actions.
When a user adds a webhook from github to deeptracy, we need to handle their pushes to launch scans for their projects.
Add firebase support to send push notifications when a scan finishes
Project should have a default language to process scans.
If a scan is launched for a project without specifying the language de default language for the project must be used.
This issue comes from the need to launch scans directly from webhooks (bitbucket or github) and not having the language available.
When building the wheel to push to pypi, avoid adding the tests package in the bundle.
Right now all scans are made in the master branch, we need to support scan in other branches as well.
Change targest names in Makefile from behave to at
Scans should be limited by time.
A single project can execute a scan only 1 per 15 mins (configurable)
All pieces has poor logs and low info on execution and development mode. We need to configure loggers with proper formatting, allowing some configuration to be taken from the environment (like the log level and destination)
We could set the requested API version on Header in order to preserve the same path across different API versions
API Docs generating crash on "make html"
When we scan a nodejs project we need to generate and save in the database a dependency list.
This list is needed to dynamic
find new vulnerabilities when they are found and published without the needing of scanning again projects.
This issue is the first step to achieve on of the main features of deeptracy, that is find vulnerabilities when they are found in libraries without the need of scaning your project
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.