carbonblack / cb-wildfire-connector Goto Github PK

The cb-wildfire-connector doesn't currently make an efficient use of the API. It supports wildfire analysis for a maximum of 4 simultaneous files at a time. If those 4 files happen to take 15 mins each (the maximum analysis time for wildfire), then other files either aren't submitted to wildfire or get backed up waiting for analysis.

The wildfire API supports bulk checking of file hashes. It's possible to submit a list of hashes to wildfire and wildfire will return the current verdict or status for each hash. This is a much more efficient call than checking each hash individually repeatedly. Carbon black could continually submit files to wildfire and add/remove the hashes for those files from the aforementioned list as verdicts are rendered. This also has the added benefit of allowing CB to keep submitting files to wildfire without waiting for one of the 4 threads to be freed up to analyze the next file.

Binaries keep getting submitted over and over again when we timeout waiting for the analysis to complete

Can we improve our connection retry logic for the connector? I have the following on a python-cb-wildfire-connector-2.5-6.x86_64 version:

2017-08-05 18:15:06,626: init: INFO: Received a network connection error from https:// : HTTPSConnectionPool(host=' ', port= ): Max retries exceeded with url: /api/info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1362d72150>: Failed to establish a new connection: [Errno 111] Connection refused',))
2017-08-05 18:15:06,626: init: INFO: Failed to connect to Cb Response Server, retrying in 30 secs...
2017-08-05 18:15:51,676: init: INFO: Received a network connection error from https:// : HTTPSConnectionPool(host=' ' ', port= ): Max retries exceeded with url: /api/info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1362d72a90>: Failed to establish a new connection: [Errno 111] Connection refused',))
2017-08-05 18:15:51,676: init: INFO: Failed to connect to Cb Response Server, retrying in 30 secs...
2017-08-05 18:16:36,726: init: INFO: Received a network connection error from https:// : HTTPSConnectionPool(host=' ', port= ): Max retries exceeded with url: /api/info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1362d72bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
2017-08-05 18:16:36,726: init: INFO: Failed to connect to Cb Response Server, retrying in 30 secs...
2017-08-05 18:17:06,757: daemon: CRITICAL: Could not connect to Cb server at https:// (ConfigurationError)
2017-08-05 18:17:06,757: daemon: CRITICAL: Traceback: Traceback (most recent call last):
File "site-packages/cbint/utils/daemon.py", line 166, in start
File "cbopensource/connectors/wildfire/bridge.py", line 182, in validate_config
File "site-packages/cbint/utils/detonation/init.py", line 172, in validate_config
ConfigurationError: Could not connect to Cb server at https://

This was during a start of services:

2017-08-05 18:22:02 [7920] ---- Starting cb-enterprise(6.1.2.170707.2323) services.... ----
2017-08-05 18:22:08 [7920] Started cb-supervisord (PID 8378)
2017-08-05 18:28:11 [7920] ---- cb-enterprise successfully started ----

Would be great if the connector didn't terminate after 2 minutes of connectivity problems, especially since instances with larger datastores in 6.1 will now take several minutes to completely build the solr index and start all services.

Customer is not interested in performing historical analyses, only "new" binaries