splunk / addonfactory-ucc-library

UCC Helper Library used by Splunk Add-ons

Splunk OCSF Extension

The Splunk schema extension repository

addonfactory-ucc-library's Issues

httplib2 to requests migration

Currently, splunktaucclib uses the httplib2 internally to make the API calls. In the next release of splunktaucclib (6.0.0), we will be replacing the httplib2 library with the requests library. This issue provides the change details.

Difference between httplib2 and requests:

1. httplib2 returns (response_headers, content) and requests returns response
2. In httplib2, the content is a type of bytes object which needs to be converted into an str object. In requests, we can directly get the str object using the response.text

Example:

Using httplib2:

import httplib2
h = httplib2.Http()
(response_headers, content) = h.request("http://example.org/", "GET")
print(response_headers.status)
data = content.decode('utf-8')

Using requests:

import requests
response = requests.request("GET", "http://example.org/")
print(response.status_code)
data = response.text

Code change require to use the splunktaucclib v6.0.0

Note: If you are using the splunktaucclib.alert_actions_base.ModularAlertBase class directly in your code then you need to make the below change in your source code to use the splunktaucclib v6.0.0.

• The implementation of build_http_connection method (of class ModularAlertBase) is removed in the splunktaucclib v6.0.0. We recommend users to directly use the requests.request function to make the API calls. Doc: https://docs.python-requests.org/en/latest/api/

UCC libs 4.1.1 fails appinspect due to permissions on some py files

Latest 4.1.1 release fails appinspect:

          "checks": [
            {
              "description": "Check that files outside of the bin/ and appserver/controllers directory do not have execute\n permissions and are not .exe files.\n On Unix platform, Splunk recommends 644 for all app files outside of the bin/ directory, 644 for\n scripts within the bin/ directory that are invoked using an interpreter (e.g. python my_script.py\n or sh my_script.sh), and 755 for scripts within the bin/ directory that are invoked directly\n (e.g. ./my_script.sh or ./my_script).\n On Windows platform, Splunk recommends removing user's FILE_GENERIC_EXECUTE for all app files outside\n of the bin/ directory except users in ['Administrators', 'SYSTEM', 'Authenticated Users', 'Administrator'].",
              "messages": [
                {
                  "code": "reporter.fail(",
                  "filename": "check_source_and_binaries.py",
                  "line": 86,
                  "message": "This file has execute permissions for owners, groups, or others. File: lib/splunktaucclib/modinput_wrapper/__init__.py",
                  "result": "failure",
                  "message_filename": "lib/splunktaucclib/modinput_wrapper/__init__.py",
                  "message_line": null
                },
                {
                  "code": "reporter.fail(",
                  "filename": "check_source_and_binaries.py",
                  "line": 86,
                  "message": "This file has execute permissions for owners, groups, or others. File: lib/splunktaucclib/modinput_wrapper/base_modinput.py",
                  "result": "failure",
                  "message_filename": "lib/splunktaucclib/modinput_wrapper/base_modinput.py",
                  "message_line": null
                }
              ],
              "name": "check_for_bin_files",
              "tags": [
                "splunk_appinspect",
                "appapproval",
                "cloud",
                "private_app"
              ],
              "result": "failure"
            }

This is due to the permissions of the following new files:

-rwxr-xr-x@ 1 guilhem  staff  21517 19 Apr 14:37 ./modinput_wrapper/base_modinput.py
-rwxr-xr-x@ 1 guilhem  staff      0 19 Apr 14:37 ./modinput_wrapper/__init__.py

These permissions should be equivalent to others Python files, example:

-rw-r--r--@ 1 guilhem  staff    165 19 Apr 14:37 ./logging_helper.py
-rw-r--r--@ 1 guilhem  staff   1996 19 Apr 14:37 ./global_config/schema.py
-rw-r--r--@ 1 guilhem  staff  11821 19 Apr 14:37 ./global_config/configuration.py
-rw-r--r--@ 1 guilhem  staff   1835 19 Apr 14:37 ./global_config/__init__.py

What on a search head would cause this error in multiple ucc-built add-ons?

I'm troubleshooting a UCC app that where the configuration screen returns a 500 error, with the below error in the splunkd log. On a hunch, I tried a different app built with UCC (the official databricks add-on) and it behaves the same way. Do you have any ideas about what we should be looking at to resolve this error?

09-14-2023 08:07:06.458 -0400 ERROR AdminManagerExternal [1499431 TcpChannelThread] - Unexpected error "<class 'splunktaucclib.rest_handler.error.RestError'>" from python handler: "REST Error [500]: Internal Server Error -- Migrating failed. Traceback (most recent call last):\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/urllib3/connectionpool.py", line 710, in urlopen\n chunked=chunked,\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/urllib3/connectionpool.py", line 449, in _make_request\n six.raise_from(e, None)\n File "<string>", line 3, in raise_from\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/urllib3/connectionpool.py", line 444, in _make_request\n httplib_response = conn.getresponse()\n File "/opt/splunk/lib/python3.7/http/client.py", line 1373, in getresponse\n response.begin()\n File "/opt/splunk/lib/python3.7/http/client.py", line 319, in begin\n version, status, reason = self._read_status()\n File "/opt/splunk/lib/python3.7/http/client.py", line 280, in _read_status\n line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")\n File "/opt/splunk/lib/python3.7/socket.py", line 589, in readinto\n return self._sock.recv_into(b)\nConnectionResetError: [Errno 104] Connection reset by peer\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/requests/adapters.py", line 449, in send\n timeout=timeout\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/urllib3/connectionpool.py", line 786, in urlopen\n method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/urllib3/util/retry.py", line 550, in increment\n raise six.reraise(type(error), error, _stacktrace)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/urllib3/packages/six.py", line 769, in reraise\n raise value.with_traceback(tb)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/urllib3/connectionpool.py", line 710, in urlopen\n chunked=chunked,\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/urllib3/connectionpool.py", line 449, in _make_request\n six.raise_from(e, None)\n File "<string>", line 3, in raise_from\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/urllib3/connectionpool.py", line 444, in _make_request\n httplib_response = conn.getresponse()\n File "/opt/splunk/lib/python3.7/http/client.py", line 1373, in getresponse\n response.begin()\n File "/opt/splunk/lib/python3.7/http/client.py", line 319, in begin\n version, status, reason = self._read_status()\n File "/opt/splunk/lib/python3.7/http/client.py", line 280, in _read_status\n line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")\n File "/opt/splunk/lib/python3.7/socket.py", line 589, in readinto\n return self._sock.recv_into(b)\nurllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/splunk_aoblib/rest_migration.py", line 19, in handle\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/splunk_aoblib/rest_migration.py", line 68, in _migrate\n self._migrate_conf()\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/splunk_aoblib/rest_migration.py", line 89, in _migrate_conf\n conf_file, stanzas = self._load_conf(conf_file_name)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/splunk_aoblib/rest_migration.py", line 175, in _load_conf\n if conf_file_name not in self.client.confs:\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/splunklib/client.py", line 1718, in __contains__\n response = self.get(key)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/splunklib/client.py", line 1668, in get\n return super(Collection, self).get(name, owner, app, sharing, **query)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/splunklib/client.py", line 766, in get\n **query)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/splunklib/binding.py", line 290, in wrapper\n return request_fun(self, *args, **kwargs)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/splunklib/binding.py", line 71, in new_f\n val = f(*args, **kwargs)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/splunklib/binding.py", line 686, in get\n response = self.http.get(path, all_headers, **query)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/splunklib/binding.py", line 1199, in get\n return self.request(url, { 'method': "GET", 'headers': headers })\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/splunklib/binding.py", line 1259, in request\n response = self.handler(url, message, **kwargs)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/solnlib/splunk_rest_client.py", line 147, in request\n **kwargs,\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/requests/api.py", line 61, in request\n return session.request(method=method, url=url, **kwargs)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/requests/sessions.py", line 542, in request\n resp = self.send(prep, **send_kwargs)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/requests/sessions.py", line 655, in send\n r = adapter.send(request, **kwargs)\n File "/opt/splunk/etc/apps/conneQt/bin/conneqt/aob_py3/requests/adapters.py", line 498, in send\n raise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))\n". See splunkd.log/python.log for more details.

Credentials encrypting not writing in the correct app

Hi,

I'm not sure if it's bug, an enhancement request or a rtfm request ^^'

We have a Modular input based on UCC generator which is used for several client.
We have configuration app which provides account/inputs for those client.

Our issue is that the credentials replacement with ******is not working as we expect : it's writing the ****** in the modular input itself instead of overwriting the configuration app files.

The result is that the password are always in clear text in our configuration.
In fact all edition of the config are written in the modular input instead of the config app.

How can we make sure the password credentials overwrite the proper files ?

__init__.py does not idenfity the version of the

Hi,

splunktaucclib does not contain any version / release idenfitier, unless I am wrong the following file:

splunktaucclib/__init__.py

Should contain the release version as a good practive, which makes things much harder.

Credentials not working when password does not exist

Hi,

Credentials are not working as expected when password.conf does not exist. I am using VT4Splunk app in Splunk Enterprise 9.0.3 (Debian GNU/Linux 11, Python 3.7). When accessing to Configuration page an ERR0005 is received with the following message:

"Unexpected error \"<class 'splunktaucclib.rest_handler.error.RestError'>\" from python handler: \"REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n  File \"SPLUNK_HOME/etc/apps/TA-virustotal-app/bin/ta_virustotal_app/aob_py3/splunktaucclib/rest_handler/handler.py\", line 124, in wrapper\n    for name, data, acl in meth(self, *args, **kwargs):\n  File \"SPLUNK_HOME/etc/apps/TA-virustotal-app/bin/ta_virustotal_app/aob_py3/splunktaucclib/rest_handler/handler.py\", line 303, in _format_response\n    masked = self.rest_credentials.decrypt_for_get(name, data)\n  File \"SPLUNK_HOME/etc/apps/TA-virustotal-app/bin/ta_virustotal_app/aob_py3/splunktaucclib/rest_handler/credentials.py\", line 203, in decrypt_for_get\n    data[field_name] = clear_password[field_name]\nTypeError: 'NoneType' object is not subscriptable\n\". See splunkd.log/python.log for more details."

• If trying to reproduce the issue and fails, delete local/password.conf and it should work.

Checking the code in credentials.py I noticed something strange. Code in here catches CredentialNotExistException and returns None. However, when using _get function, for example here, CredentialNotExistException is trying to be caught but this isn't never happening because _get already caught it.

I guess this isn't the expected behaviour. I edited the _get function not to catch the CredentialNotExistException and it works correctly. There are only two calls to _get function, here and here that don't catch that exception, but it can be fixed by adding:

try:
  original_clear_password = self._get(name)
except CredentialNotExistException:
  original_clear_password = None

The other three calls to _get function in the file do expect to catch the exception. Other solution may be not to edit the _get function and replace the except CredentialNotExistException: by if self._get(...) is None. Both solutions are valid and work as expected.

cim_actions.py is out of date

I was tracing down why ModularAction.result2stash was not working correctly with mv fields, and i noted that cim_actions.py has some problems with mv fields and with locating the info.csv file for per result alerts.

One thing i did note is that this was not an issue in some other splunk apps, and i noted that Splunk_SA_CIM has a totally different version of cim_actions.py that has fixes for both of these issues.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Pending Approval

These branches will be created by Renovate only once you click their checkbox below.

• ci(deps): pin dependencies (PySocks, pytest, requests, solnlib, splunk-add-on-ucc-framework, splunk-packaging-toolkit, splunk-sdk, splunktalib, urllib3)
• ci(deps): update dependency urllib3 to v2
• chore(deps): lock file maintenance
• 🔐 Create all pending approval PRs at once 🔐

Detected dependencies

github-actions

.github/workflows/aggreements.yaml

• splunk/addonfactory-github-workflows v1.3

.github/workflows/build-test-release.yml

• actions/checkout v3
• splunk/addonfactory-test-matrix-action v1
• actions/checkout v3
• actions/upload-artifact v3
• actions/checkout v3
• apache/skywalking-eyes v0.4.0
• actions/checkout v3
• actions/setup-python v4
• pre-commit/action v3.0.0
• actions/checkout v3
• returntocorp/semgrep-action v1
• actions/checkout v3
• actions/setup-python v4
• actions/upload-artifact v3
• actions/checkout v3
• actions/setup-python v4
• actions/checkout v3
• actions/setup-python v4
• actions/checkout v3
• actions/setup-python v4
• splunk/semantic-release-action v1.3
• splunk/pypi-publish-action v1.0

poetry

pyproject.toml

• splunktalib ^3.0.4
• requests ^2.31.0
• urllib3 <2
• PySocks ^1.7.1
• splunk-sdk >=1.6.18
• solnlib ^4.11.2
• pytest ^7.2
• splunk-add-on-ucc-framework ^5.27.1
• splunk-packaging-toolkit ^1.0.1

---

• Check this box to trigger a request for Renovate to run again on this repository

introspection error with modular input

It seems I have an older base file for my modular input, I had to comment out lines 112 and 127 in file:
splunktaucclib/modinput_wrapper/base_modinput.py

for introspection to pass.

The current output from UCC doesnt generate in this way, it might have been copypasta from add on builder.

Deleting and re-generating the modular input wrapper file using UCC also fixed the issue.

Found script "/opt/splunk/etc/apps/oversight/bin/oversight.py" to handle scheme "oversight".
setup_interpretter(): path=/opt/splunk/etc/apps/oversight/bin/oversight.py --scheme schemeName=oversight python.version=
<stderr> Introspecting scheme=oversight:  Traceback (most recent call last):
<stderr> Introspecting scheme=oversight:    File "/opt/splunk/etc/apps/oversight/bin/oversight.py", line 9, in <module>
<stderr> Introspecting scheme=oversight:      from splunktaucclib.modinput_wrapper import base_modinput as modinput_wrapper_base_modinput
<stderr> Introspecting scheme=oversight:    File "/opt/splunk/etc/apps/oversight/bin/oversight/splunktaucclib/modinput_wrapper/base_modinput.py", line 112
<stderr> Introspecting scheme=oversight:      print(traceback.format_exc(), file=sys.stderr)
<stderr> Introspecting scheme=oversight:                                        ^
<stderr> Introspecting scheme=oversight:  SyntaxError: invalid syntax
Introspecting scheme=oversight: script running failed (exited with code 1).
Unable to initialize modular input "oversight" defined in the app "oversight": Introspecting scheme=oversight: script running failed (exited with code 1)..

For reference, the older version of the file from add-on builder:

import os
import sys
import time
import datetime
import json

import modinput_wrapper.base_modinput
from solnlib.packages.splunklib import modularinput as smi


import input_module_oversight as input_module

bin_dir = os.path.basename(__file__)

"""
    Do not edit this file!!!
    This file is generated by Add-on builder automatically.
    Add your modular input logic to file input_module_oversight.py
"""


class ModInputoversight(modinput_wrapper.base_modinput.BaseModInput):
    def __init__(self):
import os
import sys
import time
import datetime
import json

import modinput_wrapper.base_modinput
from solnlib.packages.splunklib import modularinput as smi


import input_module_oversight as input_module

bin_dir = os.path.basename(__file__)

"""
    Do not edit this file!!!
    This file is generated by Add-on builder automatically.
    Add your modular input logic to file input_module_oversight.py
"""


class ModInputoversight(modinput_wrapper.base_modinput.BaseModInput):
    def __init__(self):
    ...

HTH