carbonblack / cbc-syslog Goto Github PK
View Code? Open in Web Editor NEWSyslog Connector for the Carbon Black Cloud
License: MIT License
Syslog Connector for the Carbon Black Cloud
License: MIT License
Installation fails on CentOS 7, Ubuntu 18.04, and in the included Dockerfile when following the installation instructions in the README.md
Thanks to @bigblueswope again :)
And at line 31, lets add an informative message when the API key is wrong
if response.status_code == 401:
logger.warn("Authentication failed check config file for proper Connector ID and API key")
sys.exit(1)
differentiate between Monitored events vs Threat events
executor failed running [/bin/sh -c yum update -y && yum install -y gcc python-devel && yum install -y epel-release && yum install -y python-pip rpm-build && pip install --upgrade pip && pip install pyinstaller==2.7 && mkdir -p /root/build-root /root/cbc-syslog /root/rpmbuild/SOURCES]: exit code: 1
In the Windows OS environment, the scripts (core.py, output.py etc) will result in the following error when attempting to use unsupported characters (":") in a file name on Windows.
2023-12-06 11:17:38,983 - cbc_syslog.cli - ERROR - [Errno 22] Invalid argument: 'C:\cbc-syslog2\backup\cbc-2023-12-06T11:16:08.436704Z.bck'
Traceback (most recent call last):
File "C:\Program Files\Python312\Lib\site-packages\cbc_syslog\cli.py", line 115, in main
succeeded = poll(Config(args.config_file))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python312\Lib\site-packages\cbc_syslog\core.py", line 149, in poll
with open(backup_file, "a") as backup:
^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 22] Invalid argument: 'C:\cbc-syslog2\backup\cbc-2023-12-06T11:16:08.436704Z.bck'
It always reproduces in the Windows environment.
Use file names supported by Windows.
No response
If configured to use multiple orgs/connectors, we need to be able to identify which org the notification came from. This could be done in the connector by including the ServerURL used in the API call and the ConnectorID. This would allow you to pull from multiple orgs, send to SIEM and still identify which org that notification came from.
Although we are pushing to TLS we do not want to make TLS mandatory we want to support it.
https://github.com/carbonblack/cb-defense-syslog-tls/blob/master/cb-defense-connector.py#L220
https://github.com/carbonblack/cb-defense-syslog-tls/blob/master/cb-defense-connector.py#L93
as well as others to ensure it is handled correctly.
https://github.com/carbonblack/cb-defense-syslog-tls/blob/master/cb-defense-connector.py#L248
https://server.com
server.com
https;//server.com
my thought is some simple error handling on that. that area will be very error prone.
For cbc-syslog version 1.0.2, following line seems to be wrong.
158 if not output_format == 'cef' and not output_format == 'json' and output_format == 'leef':
I think it should be:
158 if not output_format == 'cef' and not output_format == 'json' and not output_format == 'leef':
Reported by @bigblueswope
If you've defined multiple SIEM connectors, as soon as one returns no records it calls sys.exit() instead of just returning None
See https://github.com/carbonblack/cb-defense-syslog-tls/blob/master/cb-defense-syslog.py#L353 and line 368:
need to move lines 370-383 into the (not really there) else: portion of the if statement at 366
if not log_messages:
logger.info("There are no messages to forward to host")
else:
logger.info("Sending {} messages to {}:{}".format(len(log_messages),
output_params['output_host'],
output_params['output_port']))
#
# finally send the messages
#
for log in log_messages:
template = Template(config.get('general', 'template'))
template = Template(config.get('general', 'template'))
send_syslog_tls(output_params['output_host'],
output_params['output_port'],
template.render(log),
output_params['output_type'])
I've set up the cbc-syslog tool using the poll mode on my Debian server and I do receive Audit Log syslog messages in my SIEM system but no Alerts. I have the following parameters setup in my conf file:
[general]
backup_dir = "/tmp"
output_format = "template"
output_type = "udp"
udp_out = "IP:Port"
[nav]
server_url = "defense-eu.conferdeploy.net"
org_key = "xxxxx"
custom_api_id = "xxxxx"
custom_api_key = "xxxxx"
audit_logs_enabled = true
alerts_enabled = true
[[nav.alert_rules]]
type = [ "WATCHLIST", "DEVICE_CONTROL", "CB_ANALYTICS", "CONTAINER_RUNTIME", "HOST_BASED_FIREWALL", "INTRUSION_DETECTION_SYSTEM" ]
minimum_severity = 1
[alerts_template]
template = "{{datetime_utc}} localhost CEF:1|{{vendor}}|{{product}}|{{product_version}}|{{reason_code}}|{{reason}}|{{severity}}|{{extension}}"
type_field = "type"
time_format = "%b %d %Y %H:%m:%S"
time_fields = ["backend_timestamp"]
[alerts_template.extension]
default = "cat={{type}}\tact={{sensor_action}}\toutcome={{run_state}}"
CB_ANALYTICS = "cat={{type}}\tact={{sensor_action}}\toutcome={{run_state}}\tframeworkName=MITRE_ATT&CK\tthreatAttackID={{attack_tactic}}:{{attack_technique}}"
[audit_logs_template]
template = "{{datetime_utc}} localhost CEF:1|{{vendor}}|{{product}}|{{product_version}}|Audit Logs|{{description}}|1|{{extension}}"
type_field = ""
time_format = "%b %d %Y %H:%m:%S"
time_fields = ["eventTime"]
[audit_logs_template.extension]
default = "rt={{eventTime}}\tdvchost={{orgName}}\tduser={{loginName}}\tdvc={{clientIp}}\tcs4Label=Event_ID\tcs4={{eventId}}"
I have also checked with tcpdump what data is sent to the output port but also there only the audit log messages are sent.
When I use the CB API directly to test (using curl) I do get alerts returned...
Any idea why I don't get alerts out of the cbc-syslog tool using the poll mode?
Alerts and audit logs should be polled...
No additional context
When I installed cbc-syslog 1.0.2 on CentOS with python3.6, I get following error when testing:
# /usr/bin/python3.6 /usr/local/lib/python3.6/site-packages/cbc_syslog/cb_defense_syslog.py --config-file /usr/local/lib/python3.6/site-packages/cbc_syslog/root/etc/cb/integrations/cb-defense-syslog/cb-defense-syslog.conf --log-file /var/log/cb/integrations/cb-defense-syslog/cb-defense-syslog.log
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/cbc_syslog/cb_defense_syslog.py", line 5, in <module>
import ConfigParser
ModuleNotFoundError: No module named 'ConfigParser'
#
I needed following fix to make it run:
# diff -u /usr/local/lib/python3.6/site-packages/cbc_syslog/cb_defense_syslog.py.orig /usr/local/lib/python3.6/site-packages/cbc_syslog/cb_defense_syslog.py
--- /usr/local/lib/python3.6/site-packages/cbc_syslog/cb_defense_syslog.py.orig 2020-07-21 16:20:57.418225523 +0900
+++ /usr/local/lib/python3.6/site-packages/cbc_syslog/cb_defense_syslog.py 2020-07-21 16:21:22.824624882 +0900
@@ -2,7 +2,7 @@
import ssl
import sys
import argparse
-import ConfigParser
+import configparser
import requests
from jinja2 import Template
import os
@@ -35,7 +35,7 @@
"""
global config
try:
- config = ConfigParser.ConfigParser()
+ config = configparser.ConfigParser()
config.readfp(open(args.config_file))
except Exception as e:
logging.error(e, exc_info=True)
However, the README suggests single quoted strings in the documentation. Is it possible to change that?
Receive an error when trying to build the docker container:
Step 5/11 : RUN pip install -r requirements.txt && python setup.py -v bdist_binaryrpm && cd /root/build-root && rpm2cpio /root/rpmbuild/RPMS/x86_64/python-cb-defense-syslog-*.rpm | cpio -id
---> Running in 264935c8e5ee
Collecting Jinja2==2.8 (from -r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/96/a1/c56bc4d99dc2663514a8481511e80eba8994133ae75eebdadfc91a5597d9/Jinja2-2.8-py2.py3-none-any.whl (263kB)
Collecting MarkupSafe==0.23 (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/c0/41/bae1254e0396c0cc8cf1751cb7d9afc90a602353695af5952530482c963f/MarkupSafe-0.23.tar.gz
Collecting requests>=2.20.0 (from -r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)
Collecting flask==0.12.4 (from -r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/2e/48/f1936dadac2326b3d73f2fe0a964a87d16be16eb9d7fc56f09c1bea3d17c/Flask-0.12.4-py2.py3-none-any.whl (81kB)
Collecting six==1.12.0 (from -r requirements.txt (line 5))
Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests>=2.20.0->-r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.20.0->-r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting idna<2.9,>=2.5 (from requests>=2.20.0->-r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
Collecting certifi>=2017.4.17 (from requests>=2.20.0->-r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl (154kB)
Collecting itsdangerous>=0.21 (from flask==0.12.4->-r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting click>=2.0 (from flask==0.12.4->-r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
Collecting Werkzeug>=0.7 (from flask==0.12.4->-r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl (322kB)
Installing collected packages: MarkupSafe, Jinja2, urllib3, chardet, idna, certifi, requests, itsdangerous, click, Werkzeug, flask, six
Running setup.py install for MarkupSafe: started
Running setup.py install for MarkupSafe: finished with status 'done'
Found existing installation: chardet 2.2.1
Uninstalling chardet-2.2.1:
Successfully uninstalled chardet-2.2.1
Successfully installed Jinja2-2.8 MarkupSafe-0.23 Werkzeug-0.14.1 certifi-2018.11.29 chardet-3.0.4 click-7.0 flask-0.12.4 idna-2.8 itsdangerous-1.1.0 requests-2.21.0 six-1.12.0 urllib3-1.24.1
You are using pip version 8.1.2, however version 19.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
running bdist_binaryrpm
running sdist
running check
warning: sdist: manifest template 'MANIFEST.in' does not exist (using default file list)
error: venv/local/lib: No such file or directory
The command '/bin/sh -c pip install -r requirements.txt && python setup.py -v bdist_binaryrpm && cd /root/build-root && rpm2cpio /root/rpmbuild/RPMS/x86_64/python-cb-defense-syslog-*.rpm | cpio -id' returned a non-zero code: 1
Running on Debian 9
Docker version 18.09.0 build 4d60db4
test\test_data.py the URLs for the sample data are https://testserver .conferdeploy.net...
Is that by design?
Output the customizable Notification Name.
Current example script for CEF Alerts are this:
EXAMPLE_ALERT_CEF_TEMPLATE = """
[alerts_template]
template = "{{datetime_utc}} localhost CEF:1|{{vendor}}|{{product}}|{{product_version}}|{{reason_code}}|{{reason}}|{{severity}}|{{extension}}"
Would recommend this to make it explicit that this is an alert log to distinguish from the audit logs.
EXAMPLE_ALERT_CEF_TEMPLATE = """
[alerts_template]
template = "{{datetime_utc}} localhost CEF:1|{{vendor}}|{{product}}|{{product_version}}|Alert|{{reason_code}}|{{reason}}|{{severity}}|{{extension}}"
e.g. CEF:1|vendor|product|version|eventType|other fields
Just a small example recommendation as it appears customers just blindly copy the example toml file, and it make it easy to distinguish log types
No response
Hello team,
It seems the LEEF output format is a little bit broken..
It's weird because in your config file you don't mention the possibility to put this kind of output : "Output format of the data sent. Currently support json or cef formats".
But when we read the python file, there is this possibility : "logger.error('Must specify JSON, CEF , or LEEF output format')" (line 162)
Red Hat Enterprise Linux Server release 7.7 (Maipo)
python-cb-defense-syslog-1.2-12.x86_64
2020-01-02 17:15:21,634 - __main__ - INFO - Handling notifications for https://api-prod0X.conferdeploy.net
2020-01-02 17:15:21,635 - __main__ - INFO - Attempting to connect to url: https://api-prod0X.conferdeploy.net
2020-01-02 17:15:21,786 - __main__ - INFO - <Response [200]>
2020-01-02 17:15:21,787 - __main__ - ERROR - 'dict' object has no attribute 'json'
Traceback (most recent call last):
File "cb_defense_syslog.py", line 740, in <module>
File "cb_defense_syslog.py", line 657, in main
File "cb_defense_syslog.py", line 78, in parse_cb_defense_response_leef
AttributeError: 'dict' object has no attribute 'json'
We would like to use this connector to forward Carbon Black Defense logs to our Qradar instance. This last requires the logs in LEEF format to works properly..
Thank you in advance for your help on this case !
Regards,
Lucas A.
Users need to be able to specify http and https proxies
Is it possible to re retrieve notifications? Meaning, I get 2500 messages and for some reason I delete the resultant syslog file and I want to get them again.. how can I do that?
For now, I'm only to get new notifications/events, but not ask for them again.
Would be nice to be able to specify a Rotating Log file instead of a Syslog Destination in the configuration file.
Something like this perhaps?http://www.blog.pythonlibrary.org/2014/02/11/python-how-to-create-rotating-logs/
This would allow for a local log file output (instead of a syslog destination), which will enable logstash (or something similar) to pick-up the file in JSON format and deliver to various destinations.
In our case that's Rabbit/ElasticSearch
Occasionally get UTF-8->ASCII encoding problems. Investigate and fix!
Hello,
I'm trying to understand the CMD line. Should there be an executable in the path?
Line 40 in 97e4014
Running the build container as described sudo docker container run -it <containerId> /bin/bash
and checking the directory executed on the CMD
I only find the store
directory that is created on the Dockerfile.
Line 33 in 97e4014
Should the command be this instead: CMD ["cbc-syslog", "--config-file", "/etc/cb/integrations/cbc-syslog/cbc-syslog.conf", "--log-file", "/dev/stdout"]
? I ask because on line
Line 37 in 97e4014
No such file or directory
message.
Regards,
This code will be ran from unknown source ie, Centos/redhat of some version also it is more than likely to have no remnant Cb users,folders, etc on it.
cron vs daemon? thoughts?
I get following error when no alerts are available on the backend server.
If alerts were available, finishes without errors.
CentOS Linux release 7.6.1810 (Core)
python-cb-defense-syslog-1.2-12.x86_64
2019-04-08 09:00:02,631 - main - INFO - Number of files in store forward: 0
2019-04-08 09:00:02,631 - main - INFO - Found 1 Cb Defense Servers in config file
2019-04-08 09:00:02,631 - main - INFO - Handling notifications for https://api-prod05.conferdeploy.net
2019-04-08 09:00:02,631 - main - INFO - Attempting to connect to url: https://api-prod05.conferdeploy.net
2019-04-08 09:00:03,330 - main - INFO - <Response [200]>
2019-04-08 09:00:03,330 - main - INFO - successfully connected, no alerts at this time
2019-04-08 09:00:03,330 - main - INFO - There are no messages to forward to host
2019-04-08 09:00:03,330 - main - ERROR - 'NoneType' object is not iterable
Traceback (most recent call last):
File "cb_defense_syslog.py", line 740, in
File "cb_defense_syslog.py", line 675, in main
TypeError: 'NoneType' object is not iterable
Users will need to be able to easily set the TLS configuration for requests library that is used to interact with CBD - Should we expose this as a configuration option like requests_ca_bundle / various ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.