splunk / utbox

URL Toolbox (UTBox) is a set of building blocks for Splunk specially created for URL manipulation. UTBox has been created to be modular, easy to use and easy to deploy in any Splunk environments.

Splunk OCSF Extension

The Splunk schema extension repository

utbox's Issues

Empty `ut_domain` on URL Toolbox 1.8

• Splunk: 8.2.3.2
• Url Toolbox: 1.8

ut_domain is not populated with the iana list.

Weirdness parsing dyndns.org

Seeing some weird behavior when parsing dyndns.org FQDNs with ut_parse_extended and the mozilla list.

Expected Behavior:

With an FQDN like foo.bar.google.com, it correctly shows the tld, domain, subdomain and number of subdomain elements.

| makeresults
| eval url="foo.bar.google.com"
| eval list="mozilla"
| `ut_parse_extended(url, list)`
| table list url ut_tld ut_domain ut_subdomain ut_subdomain_count

Issue

Parsing foo.bar.dyndns.org, ut seems to think that the TLD is dyndns.org and the domain is bar.dyndns.org

| makeresults
| eval url="foo.bar.dyndns.org"
| eval list="mozilla"
| `ut_parse_extended(url, list)`
| table list url ut_tld ut_domain ut_subdomain ut_subdomain_count

Even weirder, with foo.go.dyndns.org, ut parses go.dyndns.org as the TLD, and foo.go.dyndns.org as the domain.

| makeresults
| eval url="foo.go.dyndns.org"
| eval list="mozilla"
| `ut_parse_extended(url, list)`
| table list url ut_tld ut_domain ut_subdomain ut_subdomain_count

Investigate mechanism to keep Mozilla Public Suffix List updated

It would be a neat improvement to enable a quick update of the Mozilla Public Suffix list from within the app.

We could ship a modular input that is disabled by default that periodically updates the public suffix list.

Rotation issues with utbox.log

Hi,
I have a SHC that the utbox.log file in $SPLUNK_HOME//var/log/splunk is getting very large (10Gb+) and causing disk space issues. Is there any log rotation setup for this file. Other TA log files in this directory rotate to a *.log.1 or such file and then get delete but that doesnt seem to happen for this file. This is really an issue since it seems that its set to debug by default with no way to change the logging level.

Mozilla ut_parse does not present ut_domain and other fields

Hi Daniel, Ian or Mayur,

could you please take a look at
[https://community.splunk.com/t5/All-Apps-and-Add-ons/Why-is-URL-Toolbox-not-working-with-mozilla-list/m-p/586630/thread-id/76192](Splunk Answer UTBox Mozilla Issue)?

It seems latest utbox v1.9.1 has a hicup on parsing out ut_domain and other fields when using the mozilla suffix list. iana and custom work, also did v1.8 with the mozilla list. Strange, with only that small codechange from 1.8 to 1.9.1.

Thanks a lot.

Add configuration mechanism for custom list that works via Splunk Lookup

Currently, supplying a custom list requires editing bin/suffix_list_custom.dat. As an administrator, I'd like to supply my custom list via a Splunk Lookup that does not require me to create a fork of the app.

Issue with ut_* fields when tld is not in lists

Hi,

We encounter an issue when we use URL Toolbox with subdomains that are not in DAT lists used by the python script.

It seems that the script truncate and merge the end of the URL instead of keeping the last string after a dot.

Here are some examples :

• test.containers.internal --> ut_subdomain = "test.containers" instead of "test", ut_domain = "int.host" instead of "containers.internal", ut_tld = "host" instead of "internal"
• test.redhat.com.localdomain --> ut_subdomain = "test.redhat.com" instead of "test.redhat", ut_domain = "localdo.com" instead of "com.localdomain", ut_tld = "com" instead of "localdomain"
• test.centos.pool.ntp.org.xxxlocal --> ut_subdomain = "test.centos.pool.ntp.org" instead of "test.centos.pool.ntp", ut_domain = "xxxl.org" instead of "org.xxxlocal", ut_tld = "org" instead of "xxxlocal"

When we add the TLD in DAT files used by the python script for the lists, it works well. Nevertheless we cannot add all possible and imaginable cases. The impact of this issue is concerning the correlation searches that does not detect the correct values.

Would it be please possible to update the python script to change this behavior when it does not find the TLD in DAT files and keep the correct values ? Or maybe is there a reason for that ?

We thank you in advance.

Best regards,

D.BRANGER