aholzel / ta-dmarc Goto Github PK

Hi,

I've done my own troubleshooting and Googling and I can't work out what I've missed.

This is installed on Splunk 8.2.1, however it's not working.

The main clue I've got is messaged about the session key not being passed to the script.

loglevel=CRITICAL file=dmarc-convertor.py line=142 message="Did not receive a session key from splunkd. Please enable passAuth in inputs.conf for this script"

Any idea what I need to do to fix this?

Hi, i am unable to start the setup.
running the latest version of splunk on windows 10.
App is not visible in apps. If i open the manage apps i can see TA-DMARC, clicking on setup throws "500 Internal Server Error"

the url also looks incorrect to me https://10.4.100.1/en-GB/manager/TA-dmarc/apps/local/TA-dmarc/setup?action=edit
if i change the url to https://10.4.100.1/en-GB/app/TA-dmarc it shows the app config page

App configuration
The "TA-dmarc" app has not been fully configured yet.

This app has configuration properties that can be customized for this Splunk instance. Depending on the app, these properties may or may not be required.

continue to app setup page gives out the error "The server encountered an unexpected condition which prevented it from fulfilling the request. Click here to return to Splunk homepage."

given below is the request id search results

2021-06-30 17:19:08,873 ERROR [60dc5a33921ab76cbc8c8] error:335 - Traceback (most recent call last):
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\cherrypy_cprequest.py", line 628, in respond
self._do_respond(path_info)
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\cherrypy_cprequest.py", line 687, in _do_respond
response.body = self.handler()
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\cherrypy\lib\encoding.py", line 219, in call
self.body = self.oldhandler(*args, **kwargs)
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\htmlinjectiontoolfactory.py", line 75, in wrapper
resp = handler(*args, **kwargs)
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\cherrypy_cpdispatch.py", line 54, in call
return self.callable(*self.args, **self.kwargs)
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\routes.py", line 383, in default
return route.target(self, **kw)
File "<C:\Program Files\Splunk\Python-3.7\lib\site-packages\decorator.py:decorator-gen-486>", line 2, in listEntities
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\decorators.py", line 40, in rundecs
return fn(*a, **kw)
File "<C:\Program Files\Splunk\Python-3.7\lib\site-packages\decorator.py:decorator-gen-484>", line 2, in listEntities
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\decorators.py", line 118, in check
return fn(self, *a, **kw)
File "<C:\Program Files\Splunk\Python-3.7\lib\site-packages\decorator.py:decorator-gen-483>", line 2, in listEntities
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\decorators.py", line 166, in validate_ip
return fn(self, *a, **kw)
File "<C:\Program Files\Splunk\Python-3.7\lib\site-packages\decorator.py:decorator-gen-482>", line 2, in listEntities
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\decorators.py", line 245, in preform_sso_check
return fn(self, *a, **kw)
File "<C:\Program Files\Splunk\Python-3.7\lib\site-packages\decorator.py:decorator-gen-481>", line 2, in listEntities
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\decorators.py", line 284, in check_login
return fn(self, *a, **kw)
File "<C:\Program Files\Splunk\Python-3.7\lib\site-packages\decorator.py:decorator-gen-480>", line 2, in listEntities
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\decorators.py", line 304, in handle_exceptions
return fn(self, *a, **kw)
File "<C:\Program Files\Splunk\Python-3.7\lib\site-packages\decorator.py:decorator-gen-475>", line 2, in listEntities
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\lib\decorators.py", line 359, in apply_cache_headers
response = fn(self, *a, **kw)
File "C:\Program Files\Splunk\Python-3.7\lib\site-packages\splunk\appserver\mrsparkle\controllers\admin.py", line 1739, in listEntities
app_name = eai_acl.get('app')
AttributeError: 'NoneType' object has no attribute 'get'

we are not able to download this app in splunkbase, so is it possible to provide the steps to get this app in downloadable format to install this app in our environment

Microsoft (again) has removed the [Preview] from their subjects, but is encoding the subjects off their DMARC reports.

Subjects look like this:

=?UTF-8?B?UmVwb3J0IERvbWFpbjogZXhhbXBsZS5jb20gU3VibWl0dGVyOiBwcm90ZWN0aW9uLm91dGxvb2suY29tIFJlcG9ydC1JRDogZGVhZGJlZWZkZWFkYmVlZmRlYWRiZWVmZGVhZA==?=

Unless decoded they cannot be parsed at the moment.

To decode the subject following code example could work:

"""E-mail subject decoder."""

import base64
import sys

SUBJECT1 = "=?UTF-8?B?UmVwb3J0IERvbWFpbjogZXhhbXBsZS5jb20gU3VibWl0dGVyOiBwcm90ZWN0aW9uLm91dGxvb2suY29tIFJlcG9ydC1JRDogZGVhZGJlZWZkZWFkYmVlZmRlYWRiZWVmZGVhZA==?="
SUBJECT2 = "Report Domain: example.com Submitter: protection.outlook.com Report-ID: deadbeefdeadbeefdeadbeefdead"


def decode_subject(subject):
    """Decode subject with different decoding."""
    if subject.startswith("=?"):
        subject_parts = subject.split('?')
        subject_base = subject_parts[3]
        return base64.b64decode(subject_base.encode(
            'utf-8')).decode(subject_parts[1])
    return subject


print(decode_subject(SUBJECT1))
print(decode_subject(SUBJECT2))
assert decode_subject(SUBJECT1) == SUBJECT2
assert decode_subject(SUBJECT2) == SUBJECT2

Hi,

After seeing this app in the .conf slides - https://conf.splunk.com/files/2019/slides/SEC1106.pdf, I am keen to get it working!
I have installed the app and I can get to the setup page and enter all the correct information.

But after that, nothing happens. The scripts don't seem to run and no mail is retrieved from my dmarc mailbox.
Is there something I am missing, how does the app actually run?

Cheers!

Hi,

I've recently upgraded to Splunk 8.1 and since then this is no longer working. If I try and enter the config / setup screen it just hangs.

I've tried removing it and installing from scratch, but get an Internal Server Error 500.

If I look at the splunkd.log I see the following:

11-29-2020 04:32:20.274 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': The script at path=/opt/splunk/etc/apps/TA-dmarc/bin/setup_handler.py has thrown an exception=Traceback (most recent call last):
11-29-2020 04:32:20.274 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': File "/opt/splunk/bin/runScript.py", line 82, in
11-29-2020 04:32:20.274 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': exec(open(REAL_SCRIPT_NAME).read())
11-29-2020 04:32:20.274 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': File "", line 42, in
11-29-2020 04:32:20.274 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': File "/opt/splunk/etc/apps/TA-dmarc/bin/splunklib/client.py", line 562
11-29-2020 04:32:20.274 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': except Exception, e:
11-29-2020 04:32:20.274 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': ^
11-29-2020 04:32:20.274 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': SyntaxError: invalid syntax
11-29-2020 04:32:20.275 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': Traceback (most recent call last):
11-29-2020 04:32:20.275 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': File "/opt/splunk/bin/runScript.py", line 82, in
11-29-2020 04:32:20.275 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': exec(open(REAL_SCRIPT_NAME).read())
11-29-2020 04:32:20.275 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': File "", line 42, in
11-29-2020 04:32:20.275 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': File "/opt/splunk/etc/apps/TA-dmarc/bin/splunklib/client.py", line 562
11-29-2020 04:32:20.275 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': except Exception, e:
11-29-2020 04:32:20.275 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': ^
11-29-2020 04:32:20.275 +1030 ERROR ScriptRunner - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/bin/runScript.py setup': SyntaxError: invalid syntax
11-29-2020 04:32:20.290 +1030 ERROR AdminManagerExternal - External handler failed with code '1' and output: ''. See splunkd.log for stderr output.
11-29-2020 04:32:20.292 +1030 ERROR SetupAdminHandler - Error while fetching url=/servicesNS/nobody/TA-dmarc/ta-dmarc/ta-dmarc_configure/main/?_strict=true;search=%20eai%3Aacl.app%3D%22%22%20OR%20eai%3Aacl.app%3D%22TA-dmarc%22

Previously we used the IMAP connector to a MS 365 mailbox. Now the basic authentication support has ended, and this will not work any more for MS 365 mailboxes.

https://docs.microsoft.com/en-us/lifecycle/announcements/exchange-online-basic-auth-deprecated

Please supply a supported method of collecting mail, e.g. using the graph API.

I'm using the O365 and assume I've set it up correctly. When it tries to fetch the messages from my Inbox there is an error generated
loglevel=ERROR file=mail-o365.py line=485 message="No folders where found: None"

I've set the Mailbox folder to "Inbox" but then also created a new top-level folder called "dmarc" (all lowercase) but the error is still the same. I didn't see anything in past issues about this one so I'm not sure where the issue might be.

Thanks,
Aaron

Microsoft started sending aggregation reports with sibjects like:
[Preview] Report Domain: example.com Submitter: protection.outlook.com Report-ID: 9ab4470a418d4e94ba721daea59xxxxx

The prefix [Preview] causes these reports not to be processed.

Splunk version 7.2.6
Install TA-dmarc version 3.5.1
Once click on setup, it shows error page, and in splunk web_service.log:
2019-04-21 21:44:22,920 INFO [5cbc73b68b7f8f087dbe50] error:323 - GET /en-US/manager/TA-dmarc-3.5.1/apps/local/TA-dmarc-3.5.1/setup 127.0.0.1 8065
2019-04-21 21:44:22,920 INFO [5cbc73b68b7f8f087dbe50] error:324 - 500 Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request.
2019-04-21 21:44:22,921 ERROR [5cbc73b68b7f8f087dbe50] error:325 - Traceback (most recent call last):
File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 606, in respond
cherrypy.response.body = self.handler()
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/htmlinjectiontoolfactory.py", line 72, in wrapper
resp = handler(*args, **kwargs)
File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 25, in call
return self.callable(*self.args, **self.kwargs)
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/routes.py", line 366, in default
return route.target(self, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 38, in rundecs
return fn(*a, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 119, in check
return fn(self, *a, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 167, in validate_ip
return fn(self, *a, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 335, in preform_sso_check
return fn(self, *a, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 374, in check_login
return fn(self, *a, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 394, in handle_exceptions
return fn(self, *a, **kw)
File "<string>", line 1, in <lambda>
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 449, in apply_cache_headers
response = fn(self, *a, **kw)
File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/admin.py", line 1715, in listEntities
app_name = eai_acl.get('app')
AttributeError: 'NoneType' object has no attribute 'get'

My colleague @geekusa and I have discovered a bug in dmarc-parser.py where it does not reset record values after each record reading. The result is that records that do not have certain fields, such as DKIM selector, get that information populated from the last record to have a value for that field.

This problem occurs because in line 142 report_recorddata = report_defaultdata report_recorddata is not a new dictionary object but a reference to report_defaultdata. Therefore, all changes to report_recorddata also change report_defaultdata. Our solution was to import copy in the header and then rewrite 142 as
report_recorddata = copy.deepcopy(report_defaultdata)

A deep copy appears to be necessary in order to copy all of the recursive dictionaries in report_defaultdata.

Environment: Single instance Splunk Enterprise 7.2.1 Free License, running on Windows Server 2016 Standard 64-bit.

Issue: TA-dmarc-master fails to install. The add-on does not appear in the Splunk App Manager's app list.

Found the following in splunkd.log:

01-25-2019 19:38:32.006 +1100 ERROR ScriptRunner - stderr from 'C:\Program Files\Splunk\bin\PYTHON.EXE C:\Program Files\Splunk\bin\runScript.py setup':  Traceback (most recent call last):
01-25-2019 19:38:32.006 +1100 ERROR ScriptRunner - stderr from 'C:\Program Files\Splunk\bin\PYTHON.EXE C:\Program Files\Splunk\bin\runScript.py setup':    File "C:\Program Files\Splunk\bin\runScript.py", line 78, in <module>
01-25-2019 19:38:32.006 +1100 ERROR ScriptRunner - stderr from 'C:\Program Files\Splunk\bin\PYTHON.EXE C:\Program Files\Splunk\bin\runScript.py setup':      execfile(REAL_SCRIPT_NAME)
01-25-2019 19:38:32.006 +1100 ERROR ScriptRunner - stderr from 'C:\Program Files\Splunk\bin\PYTHON.EXE C:\Program Files\Splunk\bin\runScript.py setup':    File "C:\Program Files\Splunk\etc\apps\TA-dmarc-master\bin\setup_handler.py", line 36, in <module>
01-25-2019 19:38:32.006 +1100 ERROR ScriptRunner - stderr from 'C:\Program Files\Splunk\bin\PYTHON.EXE C:\Program Files\Splunk\bin\runScript.py setup':      import classes.custom_logger as c_logger
01-25-2019 19:38:32.006 +1100 ERROR ScriptRunner - stderr from 'C:\Program Files\Splunk\bin\PYTHON.EXE C:\Program Files\Splunk\bin\runScript.py setup':    File "C:\Program Files\Splunk\etc\apps\TA-dmarc-master\bin\classes\custom_logger.py", line 39, in <module>
01-25-2019 19:38:32.006 +1100 ERROR ScriptRunner - stderr from 'C:\Program Files\Splunk\bin\PYTHON.EXE C:\Program Files\Splunk\bin\runScript.py setup':      splunk_info = si.Splunk_Info(sessionKey="NA")
01-25-2019 19:38:32.006 +1100 ERROR ScriptRunner - stderr from 'C:\Program Files\Splunk\bin\PYTHON.EXE C:\Program Files\Splunk\bin\runScript.py setup':    File "C:\Program Files\Splunk\etc\apps\TA-dmarc-master\bin\classes\splunk_info.py", line 54, in __init__
01-25-2019 19:38:32.006 +1100 ERROR ScriptRunner - stderr from 'C:\Program Files\Splunk\bin\PYTHON.EXE C:\Program Files\Splunk\bin\runScript.py setup':      log_level = self.get_config(custom_conf_file, 'main', 'log_level')
01-25-2019 19:38:32.006 +1100 ERROR ScriptRunner - stderr from 'C:\Program Files\Splunk\bin\PYTHON.EXE C:\Program Files\Splunk\bin\runScript.py setup':    File "C:\Program Files\Splunk\etc\apps\TA-dmarc-master\bin\classes\splunk_info.py", line 188, in get_config
01-25-2019 19:38:32.006 +1100 ERROR ScriptRunner - stderr from 'C:\Program Files\Splunk\bin\PYTHON.EXE C:\Program Files\Splunk\bin\runScript.py setup':      if int(active_config) > 0 and int(active_config) < 20:
01-25-2019 19:38:32.006 +1100 ERROR ScriptRunner - stderr from 'C:\Program Files\Splunk\bin\PYTHON.EXE C:\Program Files\Splunk\bin\runScript.py setup':  TypeError: int() argument must be a string or a number, not 'NoneType'
01-25-2019 19:38:32.025 +1100 ERROR AdminManagerExternal - External handler failed with code '1' and output: ''.  See splunkd.log for stderr output.
01-25-2019 19:38:32.025 +1100 ERROR SetupAdminHandler - Error while fetching url=/servicesNS/nobody/TA-dmarc-master/ta-dmarc/ta-dmarc_configure/main/?_strict=true;search=%20eai%3Aacl.app%3D%22%22%20OR%20eai%3Aacl.app%3D%22TA-dmarc-master%22

Please advise how to make it install.

Thanks,
Zoltan