homas / ioc2rpz Goto Github PK

git.zeropw.com:443 was converted to 128.443.git.zeropw.com.rpz-ip.phishtank.ioc2rpz rule

1. Check a regex which decide what kind of indicator we got for mixed feeds (probably the issue because the regex accepts IPv6)
2. Enforce fqdn/ip validation.

Why I can't dig my RPZ zone? It keep prompt ;; Couldn't verify signature: tsig indicates error

rpz config:

dig result:

I have configured the ioc2rpz and ioc2rpz.gui following the instruction. Now whenever I try to hit the REST API's it always results into failure. I have tried with all the TSIG keys configured, but always getting the same error response.
I don't know if its a bug or I am missing something, I need immediate help on this hence I am posting here. Any kind of help would be highly appreciated.

Here is the curl verbose output:

• Trying XX.XXX.XXX.XXX:8443...
• TCP_NODELAY set
• Connected to XX.XXX.XXX.XXX (XX.XXX.XXX.XXX) port 8443 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• successfully set certificate verify locations:
• CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• TLSv1.3 (IN), TLS handshake, Server hello (2):
• TLSv1.2 (IN), TLS handshake, Certificate (11):
• TLSv1.2 (IN), TLS handshake, Server key exchange (12):
• TLSv1.2 (IN), TLS handshake, Server finished (14):
• TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
• TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
• TLSv1.2 (OUT), TLS handshake, Finished (20):
• TLSv1.2 (IN), TLS handshake, Finished (20):
• SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
• ALPN, server accepted to use h2
• Server certificate:
• subject: C=xx; ST=xxx; L=xxx; O=xxxxxx; OU=xxx; CN=xxxxxx.com; emailAddress=[email protected]
• start date: Aug 22 05:29:52 2019 GMT
• expire date: Aug 21 05:29:52 2020 GMT
• issuer: C=xx; ST=xx; L=xxx; O=xxxxx; OU=xx; CN=xxxxx.com; emailAddress=[email protected]
• SSL certificate verify result: self signed certificate (18), continuing anyway.
• Using HTTP2, server supports multi-use
• Connection state changed (HTTP/2 confirmed)
• Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
• Server auth using Basic with user 'xxxxxxxxxxx'
• Using Stream ID: 1 (easy handle 0x556ceabd3540)

GET /api/v1.0/feed/dns-bh.ioc2rpz HTTP/2
Host: XX.XXX.XXX.XXX:8443
Authorization: Basic YXdzX21nbW50X2tleV8yOkI2UjBTc3cxcjBxVzQ1eGJqR0RhUU85NngzN1pjT1g5L0JqckF3Qkt5ejQ9
User-Agent: curl/7.65.1
Accept: text/plain

• Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
  < HTTP/2 401
  HTTP/2 401
  < content-length: 48
  content-length: 48
  < date: Fri, 23 Aug 2019 10:55:02 GMT
  date: Fri, 23 Aug 2019 10:55:02 GMT
  < server: Cowboy
  server: Cowboy
• Authentication problem. Ignoring this.
  < www-authenticate: Basic
  www-authenticate: Basic

<
{status: "error", msg: "Authentication failed"}

• Connection #0 to host XX.XXX.XXX.XXX left intact

subj

ioc2rpz process crashes if a shell script returns data in unicode e.g. IDN w/o conversion to punycode
E.g.

zzjpoqarqc.duckdns.org
zzqqw1.godaddysites.com
zzqsdlgmlq.duckdns.org
zzzzzzzzzzzzzzz.hyperphp.com
имяенн.010.рус

The error

exit value {badarg,[{erlang,list_to_binary,
.....
file,"/opt/ioc2rpz/src/ioc2rpz_conn.erl"},{line,79}]}

list_to_binary doesn't support unicode. So I need to fix that code:

%IOCs are provided by a local script
get_ioc(<<"shell:",CMD/binary>> = _URL, _Retry) ->
  {ok, list_to_binary(os:cmd(binary_to_list(CMD)))};

ioc2rpz logs that the zone transfer went fine. However, the client throws errors.

Server logs:

Oct 21 00:56:40 rpz2 c7735b7e2425[957]: ioc2rpz tcp6_sup child started
Oct 21 00:56:40 rpz2 c7735b7e2425[957]: CEF:0|ioc2rpz|ioc2rpz|1.1.2.3-2020123101|000202|DNS Query|3|src=10.0.0.9 spt=34613 proto=tcp qname="malware-bl.ioc2rpz" qtype="AXFR" qclass="IN" tsigkey="tkey_12345."
Oct 21 00:56:40 rpz2 c7735b7e2425[957]: Found Key ... Good timestamp ... Valid MAC
Oct 21 00:56:40 rpz2 c7735b7e2425[957]: CEF:0|ioc2rpz|ioc2rpz|1.1.2.3-2020123101|000201|RPZ transfer success|3|src=10.0.0.9 spt=34613 proto=tcp qname="malware-bl.ioc2rpz" qtype="AXFR" qclass="IN"  tsigkey="tkey_12345." transfer_time=0

However the client complains:

Oct 21 00:34:29 host.domain.com pdns-recursor[147843]: Packet (malware-bl.ioc2rpz|#251) has a TSIG record in an invalid position.

And when using dig:

user@host:~$ dig +tcp @10.0.0.5 -y hmac-sha256:tkey_12345:******= malware-bl.ioc2rpz AXFR | tail -n 8
malware-bl.ioc2rpz. 604800  IN      SOA     rpz.domain.com. me.domain.com. 1666304940 86400 3600 2592000 7200
tkey_12345.             0       ANY     TSIG    hmac-sha256. 1666306180 300 32 ****= 8958 NOERROR 0
;; Query time: 0 msec
;; SERVER: 10.0.0.5#53(10.0.0.5) (TCP)
;; WHEN: Fri Oct 21 00:49:40 CEST 2022
;; XFR size: 3 records (messages 1, bytes 299)
;; WARNING -- Some TSIG could not be validated

I have other zones using the exact same TSIG key just fine, same ioc2rpz server and same clients (2 clients both complaining of the same issue)

SOA in AXFR (correct)
local.ioc2rpz. 604800 IN SOA ioc2rpz-srv1.ioc2rpz.net. ioc2rpz.ioc2rpz.com. 1666154700 86400 900 2592000 7200
vs
SOA request (incorrect)
local.ioc2rpz. 604800 IN SOA ioc2rpz-srv1.ioc2rpz.net. ioc2rpz.ioc2rpz.com. 1666154700 7200 3600 259001 7200

Check mixed zones updates. Probably an issue with incremental updates.
incsupdate.tumblr.com.phishtank.ioc2rpz. 900 IN CNAME . *.incsupdate.tumblr.com.phishtank.ioc2rpz. 900 IN CNAME . quickfollowers.net.phishtank.ioc2rpz. 900 IN CNAME . *.quickfollowers.net.phishtank.ioc2rpz. 900 IN CNAME . 32.78.44.180.107.rpz-ip.phishtank.ioc2rpz. 230400 CLASS256 TYPE1280 \# 256 0A61756E74652D6D616461C0F50005000100000384000100012AC1FB 00050001000003840001001870707265766965772D7765627363726D 376F61366535397006636E616F7661C0710005000100000384000100 012AC222000500010000038400010014636F6E6669726D2D796F7572 2D6163636F756E7411726F736564616C657061726B6E6F727468C071 0005000100000384000100012AC25E00050001000003840001000670 617970616C0F616964616E31323334353637383938047265706C0263 6FC0330005000100000384000100012AC2A100050001000003840001 000E64737266647A696E736D6B6465770B63726561746F726C696E6B C0A10005 . 2214592768 CH URI 49884 5 "\000\001\000\000\003\132\000\001\000\020outlookadministartor\194\235\000\005\000\001\000\000\003\132\000\001\000\001*\195\019\000\005\000\001\000\000\003\132\000\001\000\010god-marine\192\245\000\005\000\001\000\000\003\132\000\001\000\001*\195D\000\005\000\001\000\000\003\132\000\001\000\028f68616e67696e67666c6f27login\192\245\000\005\000\001\000\000\003\132\000\001\000\001*\195k\000\005\000\001\000\000\003\132\000\001\000\010gon-macona\192\245\000\005\000\001\000\000\003\132\000\001\000\001*\195\164\000\005\000\001\000\000\003\132\000\001\000\012webex-secure\192q\000\005\000\001\000\000\003\132\000\001\000\001*\195\203\000\005\000\001\000\000\003\132\000\001\000\014briandesmarais\192q\000\005\000\001\000\000\003\132\000\001\000\001*\195\244\000\005\000\001\000\000\003\132\000\001\000\009morfil-fm\006blogcu" com.phishtank.ioc2rpz. 900 IN CNAME . *.morfil-fm.blogcu.com.phishtank.ioc2rpz. 900 IN CNAME . secure.runescape.com-xl.ru.phishtank.ioc2rpz. 900 IN CNAME . *.secure.runescape.com-xl.ru.phishtank.ioc2rpz. 900 IN CNAME .

begin-effard.firebaseapp.com.phishtank.ioc2rpz. 900 IN CNAME . *.begin-effard.firebaseapp.com.phishtank.ioc2rpz. 900 IN CNAME . tyrannisesprices.net.phishtank.ioc2rpz. 900 IN CNAME . *.tyrannisesprices.net.phishtank.ioc2rpz. 900 IN CNAME . www.postfinance-checkout.ch.phishtank.ioc2rpz. 900 IN CNAME . *.www.postfinance-checkout.ch.phishtank.ioc2rpz. 900 IN CNAME . 32.189.177.206.116.rpz-ip.phishtank.ioc2rpz. 230400 CLASS256 TYPE1280 \# 256 037777771064616373616E6461696C6F6370686174C03A0005000100 000384000100012AD8B200050001000003840001000767696E2D756E 61C1590005000100000384000100012AD8E300050001000003840001 0002333201340239380331323603313135D88E000500010000038400 010005626C6F627302636F02696CC03E000500010000038400010001 2AD924000500010000038400010011616C6973616E74796C6F617235 736664640674756D626C72C0F00005000100000384000100012AD94C 00050001000003840001000233320331333303313330033130330331 3736D88E00050001000003840001000A6F6D652D646F6E657261C159 00050001 ;; Got bad packet: bad label type 16532 bytes 9c 8e 80 a0 00 01 02 b9 00 00 00 01 09 70 68 69 .............phi 73 68 74 61 6e 6b 07 69 6f 63 32 72 70 7a 00 00 shtank.ioc2rpz.. fc 00 01 13 76 65 72 69 66 69 6b 61 73 69 2d 61 ....verifikasi-a 63 63 6f 75 6e 74 74 07 77 65 62 6e 6f 64 65 03 ccountt.webnode. 63 6f 6d 09 70 68 69 73 68 74 61 6e 6b 07 69 6f com.phishtank.io 63 32 72 70 7a 00 00 05 00 01 00 00 03 84 00 01 c2rpz...........

I tried to explore abit your tool :) and notice, that the publish option most of the time wouldn't work on the active Custom DNS IOC2RPZ. (Had to restart it manually)

subj

An empty line (just next line 0a) in beginning of source breaks parsing the list of indicators

subj

Events info <0.560.0> is dead. Got <0.560.0> in the config. <0.562.0> is waiting... continuously generated on a high speed.
Filling up the logs. Have to be handled.

torstatus.blutmagie.de is referenced in the cfg/ioc2rpz.conf and README.md but went offline in 2018. probably need to replace that with PAN or another reference

1. Bad full zone update

2019-09-10T04:41:29.231269+00:00 Zone "notracking.ioc2rpz" serial 1568047260, refresh time 604800 current status ready
2019-09-10T04:41:29.231634+00:00 Updating zone "notracking.ioc2rpz" full
2019-09-10T04:41:29.231856+00:00 Source  "notracking_hosts" was expired in cache
2019-09-10T04:41:29.457620+00:00 Source: "notracking_hosts", size: 2.25/MB (2358005), MD5: "d7768fc5f62ceff4c3feaa92a056c845"
2019-09-10T04:41:29.634436+00:00 Source: "notracking_hosts", got 41487 indicators, clean time 0
2019-09-10T04:41:29.685434+00:00 Source  "notracking_domains" was expired in cache
2019-09-10T04:41:30.289340+00:00 Source: "notracking_domains", size: 5.33/MB (5592314), MD5: "fd98ddd2a22f3563feba8c1cf26ad5c0"
2019-09-10T04:41:30.722246+00:00 Source: "notracking_domains", got 92379 indicators, clean time 0
2019-09-10T04:41:31.082647+00:00 Delete old records from zone "notracking.ioc2rpz".  before 133964 after 1033
2019-09-10T04:41:33.940637+00:00 Live zone "notracking.ioc2rpz", 267732 rules, 133866 IOCs
2019-09-10T04:41:33.940923+00:00 Zone "notracking.ioc2rpz" updated in 0 seconds, new serial 1568090460, 267732 rules, 133866 indicators.

2. Incremental update

2019-09-11T04:41:29.231298+00:00 Start incremental update Zone "notracking.ioc2rpz" serial 1568133660 full refresh time 43200, Ctime 1568176889 cache <<"true">> status ready
2019-09-11T04:41:29.231634+00:00 Process PID <0.17606.7> incremental update "notracking.ioc2rpz" started
2019-09-11T04:41:29.258359+00:00 Updating zone "notracking.ioc2rpz" inc. Last IXFR update 43200 seconds ago, last non-zero update 43200 seconds ago
2019-09-11T04:41:30.496608+00:00 Source: "notracking_hosts", size: 2.25/MB (2359116), MD5: "3c06772a834447f362aff0f04528a33e"
2019-09-11T04:41:30.653933+00:00 Source: "notracking_hosts", got 41505 indicators, clean time 0
2019-09-11T04:41:31.515626+00:00 Source: "notracking_domains", size: 5.34/MB (5594792), MD5: "7d5690dabc07983e470107731795948c"
2019-09-11T04:41:31.935628+00:00 Source: "notracking_domains", got 92424 indicators, clean time 0
2019-09-11T04:41:32.096719+00:00 Fetching zone "notracking.ioc2rpz" from ets
2019-09-11T04:41:32.201353+00:00 Rebuilding AXFR zone "notracking.ioc2rpz". New IOCs 132951
2019-09-11T04:41:32.225914+00:00 Zone "notracking.ioc2rpz", # of rules 2066, # of IOCs 1033
2019-09-11T04:41:32.226278+00:00 AXFR zone "notracking.ioc2rpz" was rebuilded. 2066 rules 1033 indicators. Parsed 133929 indicators.
2019-09-11T04:41:32.235334+00:00 Zone "notracking.ioc2rpz" records before 1033 after 1033.
2019-09-11T04:41:32.235449+00:00 Process PID <0.17606.7> incremental update "notracking.ioc2rpz" finished in 0 seconds

Steps

1. Fill /opt/ioc2rpz/cfg/whitelist1.txt with "yellowcabnc.com"
2. Add whitelist into RPZ with IOC Source it blocks "yellowcabnc.com"
3. Publish configuration
4. Export BIND configuration
5. Update your BIND 9 server
6. You use dig or connect to BIND 9 from your PC and you do http request to "yellowcabnc.com". The site is down with NXDOMAIN response

How can I do than whitelist allows me to do http request to "yellowcabnc.com"?

subj to validate.

Thank you for a great piece of software.

Revisiting the IOC2RPZ server I built last year I'm now wondering if port 8443 can be bound to localhost or a specific IP address instead of binding to any available IP addresses. I'm running IOC2RPZ.gui on the same server so have no need to expose the REST API interface outside of the server.

I think the docker run command can be changed to add an IP address like so: -p 172.17.0.1:8443:8443 when creating the image. Not wanting to recreate the image I tried editing the hostconfig.json but the IP address is removed from there when the image is restarted after the edit. This link mentions that config.v2.json should also be modified but there are no mappings listed under Ports.

config.v2.json:

[...]
"Ports":null,
[...]

Any suggestions?

In fact, I'd like to hide the GUI (TCP:443) behind a reverse proxy too so I can enable LDAP auth for the interface. Thus I need to stop IOC2RPZ.gui from listening on all interfaces. And I have the same issue there.

Hi

Now another observation:
I have two bind servers (9.10.3-P4-Debian) and two powerdns-recursors (debian 4.1.11) that query the ioc2rpz server.

I have the SOA time for a test RPZ zone set to 60 seconds. Then I see how all 4 servers cleanly report every minute for an IXFR transfer. But as soon as I reload the ioc2rpz configuration

curl -i -u "tkey_mgmt_1:XXXXXXXXXXXXX==" --insecure -H "Accept: text/plain" https://127.0.0.1:8443/api/mgmt/reload_cfg

I see at most one more query (IXFR) of the PDNS servers and after that there is no more communication with the ioc2rpz.

After that I have to restart the service for the pdns-recursor on both pdns to start the communication again. The two BIND servers seem to continue doing this without any problem.

Do you have an idea what this could be?

Here is the recursor.lua config and the bind config:

rpzMaster("xxx.xx.1xx.123", "sXXXXX-test.ioc2rpz", { tsigname="querykey", tsigalgo="hmac-md5", tsigsecret="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=="})

zone "sXXXX-test.ioc2rpz" {
        type slave;
        file "/etc/bind/zones/black/sXXXX-test.ioc2rpz";
        masters {xxx.xx.1xx.123 key "querykey";};
        allow-query { any; };

        masterfile format text;
};

http://data.iana.org/TLD/tlds-alpha-by-domain.txt

ioc2rpz wouldn't restart after a reboot of the server it was running on. It turns out that if a source does not return any usable data after the regex filter is applied. ioc2rpz throws a tantrum and crashes. The last log line before the crash dump indicates which source is the culprit, but it doesn't help identify why there was no content. Nor is it helpful to have the docker image crash as that takes all the other RPZ feeds down as well.

Maybe, ioc2rpz could ignore the issue with the source, disable the RPZ and log an error in the logs. If ioc2rpz.gui could subsequently show an alert, that would be the cherry on top.

That said. I can't work out why this source isn't returning anything, other than that it's using tabs instead of spaces.

https://doc.powerdns.com/recursor/changelog/4.3.html
PowerDNS/pdns#8778
https://github.com/PowerDNS/pdns/pull/9048/files/052ac720275dbb3f364b62c9bccf4753d49598d5

Check performance impact- update docs or low case each indicator instead of source

ioc2rpz does full zone transfer on IXFR requests from pdns-recursor

I didn't know where to contact you, but it apears that your website "https://ioc2rpz.net/" ist down(?) or at least not reachable from my location (switzerland).

subj

;; Warning: Message parser reports malformed message packet.

I'm trying to make ioc2rpz work with pdns recursors for IXFR rpz updates.
Unfortunately pdns recursor is not getting any updates - sees new zone but it's empty.
I have checked results with dig and they are empty too.

I don't know if that's related (I don't know erl) but I saw that ./db directory is always empty even if I change include/ioc2rpz.hrl variable SaveETS to true.

ioc2rpz is build from scratch from master branch.

ioc2rpz log after adding new line (not present before) to test.list source file:

loading hot sources []
Start incremental update Zone "test.rpz" serial 1637788440 full refresh time 60, Ctime 1637788512 cache <<"true">> status ready
Process PID <0.1217.0> incremental update "test.rpz" started
Updating zone "test.rpz" inc. Last IXFR update 60 seconds ago, last non-zero update 60 seconds ago
Source: "test", size: 2.73/MB (2858273), MD5: "5cfad58ebb171b53688713f9a66dd47e"
Source: "test", got 67817 indicators, clean time 0
Memory total 1420.3366088867188 before garbage collector. processes 231.7139129638672 binary 388.9349365234375
Memory total 1411.8892288208008 after garbage collector. processes 226.90637969970703 binary 385.6593704223633
Fetching zone "test.rpz" from ets
Finding new or updated records
Update ets. New 36806, DB 36805, Delta 1
Rebuilding AXFR zone "test.rpz". New IOCs 1
Zone "test.rpz", # of rules 73612, # of IOCs 36806
AXFR zone "test.rpz" was rebuilded. 73612 rules 36806 indicators. Parsed 36806 indicators.
Zone "test.rpz" records before 36805 after 36806.
Process PID <0.1217.0> incremental update "test.rpz" finished in 0 seconds
ioc2rpz tcp6_sup child started
CEF:0|ioc2rpz|ioc2rpz|1.2.0.0-2021073101|000202|DNS Query|3|src=172.17.0.1 spt=46270 proto=tcp qname="test.rpz" qtype="IXFR" qclass="IN" tsigkey="tkey_1."
Found Key ... Good timestamp ... Valid MAC
Zone "test.rpz", 0 rules, 0 IOCs
CEF:0|ioc2rpz|ioc2rpz|1.2.0.0-2021073101|000201|RPZ transfer success|3|src=172.17.0.1 spt=46270 proto=tcp qname="test.rpz" qtype="IXFR" qclass="IN"  tsigkey="tkey_1." transfer_time=4175

dig results:

; <<>> DiG 9.17.20-1+ubuntu20.04.1+isc+2-Ubuntu <<>> @127.0.0.1 -p5555 -y hmac-md5 test.rpz ixfr=1637788440
; (1 server found)
;; global options: +cmd
test.rpz.              604800  IN      SOA     test.testrpz.local. support.testrpz.local. 1637820240 120 60 60 60
test.rpz.              604800  IN      SOA     test.testrpz.local. support.testrpz.local. 1637788440 120 60 60 60
test.rpz.              604800  IN      SOA     test.testrpz.local. support.testrpz.local. 1637820240 120 60 60 60
test.rpz.              604800  IN      SOA     test.testrpz.local. support.testrpz.local. 1637820240 120 60 60 60
tkey_1.                 0       ANY     TSIG    hmac-md5.sig-alg.reg.int. 1637833043 300 16 [cut] 34619 NOERROR 0
;; Query time: 4124 msec
;; SERVER: 127.0.0.1#5555(127.0.0.1) (TCP)
;; WHEN: Thu Nov 25 10:37:27 CET 2021
;; XFR size: 4 records (messages 1, bytes 399)

ioc2rpz config:

{source,{"test","https://homeserver.local/rpztest/test.list","[:AXFR:]","^([A-Za-z0-9][A-Za-z0-9\-\._]+)"}}.
{rpz,{"test.rpz",120,60,60,60,"true","true","nodata",["tkey_1"],"fqdn",604800,60,["test"],[],[]}}.

Hi Vadim

If I use redirect_domain or local_cname (example: redirect_url=www.google.com) for a testdomain.com the result in dig looks like this:

;; ANSWER SECTION:
testdomain.com.              5       IN      CNAME   www.google.com.testrpz.ioc2rpz.

I tried redirect_url=www.google.com. but then, the zone file is not transfered anymore. No error output in syslog.

Compiled the latest version from your git repo and created a docker container manually.

For those interested in automatically restarting ioc2rpz

Create a new watcher restart service:

sudo systemctl edit --force --full ioc2rpz-cfg-watcher.service

[Unit]
Description=ioc2rpz restarter
After=network.target
StartLimitIntervalSec=10
StartLimitBurst=5

[Service]
Type=oneshot
ExecStart=/usr/bin/docker compose -f /opt/ioc2rpz.dc/docker-compose.yml restart ioc2rpz

[Install]
WantedBy=multi-user.target

Create a new watcher path service:

sudo systemctl edit --force --full ioc2rpz-cfg-watcher.path

[Path]
Unit=ioc2rpz-cfg-watcher.service
PathChanged=/opt/ioc2rpz/cfg/ioc2rpz.conf

[Install]
WantedBy=multi-user.target

Enable and start the new services:

sudo systemctl enable --now ioc2rpz-cfg-watcher.{path,service}

Check the new services with:

sudo systemctl status ioc2rpz-cfg-watcher
sudo systemctl status ioc2rpz-cfg-path

Now edit the config file and observe whether ioc2rpz is restarted

sudo journalctl -fu docker

And then edit the config file, by changing the time stamp in the first line of the config file (the comment).

If a URL was removed from a source an empty binary string (<<>>) is saved in the config. It is not correctly handled when RPZ is updated.

Hello developers,
I have installed the software using docker images and have the AXFR working with a TSIG key. But when I do the https request with the same key, I'm getting the following error.
`# curl -i -u "tkey-3lu9dnuymu-89272zesfa:QVeIrOj5v+7Gk2NNyzpLxLEBlacXQ/KBASHMCGQfTKI=" --insecure https://192.168.0.2:8443/api/v1.0/feed/notracking.ioc2rpz
HTTP/1.1 401 Unauthorized
content-length: 48
date: Tue, 20 Apr 2021 03:41:08 GMT
server: Cowboy
www-authenticate: Basic

{status: "error", msg: "Authentication failed"}`

I thought I had it all working, then come back a few days later to find RPZ not working in PowerDNS-recursor. This is the error log I get:

Dec 17 19:38:37 rdns1 pdns-recursor[3684822]: msg="Unable to load RPZ zone, will retry" subsystem="rpz" level=1 prio=4 ts="1639766317.724" exception="Packet (gambling.ioc2rpz|#252) has a TSIG record in an invalid position." from="10.1.1.1" refresh="10" zone="gambling.ioc2rpz"

What on earth does "TSIG record in an invalid position" mean?

Check if an IOC can be used in RPZs
ioc2rpz_conn:clean_feed

Hi Homas

I have updated my ioc2rpz installation to the latest version (not dev, but master) Some parts of the configuration could be read again, some parts I had to rebuild.

But what I noticed: If I use "Local records" with "local_domain=test.ch" for a RPZ stream, the zones look like this:

dig @8x.xx.xx.xx sub.example1.com

; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 16106
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;sub.example1.com.              IN A

;; ANSWER SECTION:
sub.example1.com.       5 IN CNAME test\.ch

A backslash is inserted before each point. Even if I do this with a target subdomain
refused.test.ch --> refused\.test\.ch

Is the problem already known?

Config line:
{rpz,{"xyz-test.ioc2rpz",60,3600,2592000,7200,"true","true",[{"local_cname","test.ch"}],["querykey"],"fqdn",60,60,["local_blacklist_test"],[],["whitelist_xyz_global"]}}.

Which REGEX shorthand expressions are supported? I had initially used /d for digits and \t for tabs, but these aren't working. \d is easily replaced with [0-9] however how should one match a tab?

However, I'm now wondering if my issue isn't that, but something else. I can't seem to get ioc2rpz to read any entries from the list I've got. To test I removed the tabs from the text file to see if the REGEX I had failed to match lines with tabs in it. I then simplified the REGEX, but still I can't get ioc2rpz to accept any of the domains listed in the source.

Dec  9 17:09:19 rpz2 c7735b7e2425[1525827]: Source: "Belgium_Gambling_Commission_BL", size: 17.53/KB (17951), MD5: "b3d707807d134c9e5f4e65d23e94ba50" 
Dec  9 17:09:19 rpz2 c7735b7e2425[1525827]: Source: "Belgium_Gambling_Commission_BL", got 0 indicators, clean time 0 

A sample of the records as contained in a simple text file hosted on a nearby web server is given below. The date is a date as listed on the website I'm scraping these domains from, they are not expiry dates. Could it be that they are interpreted as such? What am I doing wrong?

bingoround.com         2012-02-16 00:00:00         # 1
myglobalgames.com         2012-02-16 00:00:00         # 2
titanpoker.com         2012-02-16 00:00:00         # 3
jackpotcity.com         2012-02-16 00:00:00         # 4
casino.com         2012-02-16 00:00:00         # 5

The regex I'm using to read these values: ^([A-Za-z0-9][A-Za-z0-9\-\._]+)\ *[12][\d-]{9}\ [\d:]{8}\ *#\ [0-9]+$

In the documentation it is presented the following actions:

Action. Supported actions: nxdomain, nodata, passthru, drop, tcp-only, {"redirect_domain","example.com"}, {"redirect_ip","127.0.0.1"} and list of local records [{"local_aaaa","fe80::1"}, {"local_a","127.0.0.1"}, {"local_cname","www.example.com"}, {"local_txt","Text Record"}]. redirect_domain is an alias for local_cname. redirect_ip is an alias for local_a, local_aaaa;

Exploring ipc2orzp-gui it was not possible to perform the following actions:

• {"redirect_domain","example.com"} - No option
• {"redirect_ip","127.0.0.1"} - No option
• local records - Not parsing correctly

In regards to the actions label has No option the ioc2rpz-gui doesn't currently allow it.

Abording the action Local Records throught ioc2rpz-gui its identified a parsing issue presented bellow:

local_cname www.test.com -> On config file appears has: ...,[{"local_cname www.test.com", ""},...

"Warning: Message parser reports malformed message packet." reported by dig if a zone consists of multiple packets.

We see the following error in case if the connection was terminated on the other side (e.g. FW). We need to implement "nice error handling".
16:48:14 exception error: no match of right hand side value {error,closed}
16:48:14 in function ioc2rpz:send_dns_tcp/3 (src/ioc2rpz.erl, line 88)
16:48:14 in call from ioc2rpz:send_cached_zone/7 (src/ioc2rpz.erl, line 538)
16:48:14 in call from ioc2rpz:parse_dns_request/3 (src/ioc2rpz.erl, line 208)
16:48:14 in call from ioc2rpz:handle_info/2 (src/ioc2rpz.erl, line 58)
16:48:14 in call from gen_server:try_dispatch/4 (gen_server.erl, line 637)
16:48:14 in call from gen_server:handle_msg/6 (gen_server.erl, line 711)
16:48:14 in call from gen_server:handle_msg/6 (gen_server.erl, line 711)
16:48:14 ancestors: [ioc2rpz_sup,<0.75.0>]

Apparently in OTP 23 a bunch of crypto functions were deprecated and in OTP 24 those functions were removed. Judging by the errors I'm seeing I think the HMAC code needs to be adapted to work with OTP 24 and later. The docs indicate that there's a new crypto API.

Looks like the code needs to be updated from:

• crypto:hmac(md5,TSIG#dns_TSIG_RR.key,PKT);

To

• :crypto.mac(:hmac, :md5, TSIG#dns_TSIG_RR.key,PKT)

Below is the errors I'm seeing:

Sep 22 12:49:49 rpzmasterse01 ioc2rpz: ** {'function not exported',
Sep 22 12:49:49 rpzmasterse01 ioc2rpz: [{crypto,hmac,
Sep 22 12:49:49 rpzmasterse01 ioc2rpz: [sha512,

From https://erlang.org/download/OTP-24.0.README
OTP-16656 Application(s): crypto

           The functions and cipher names that were deprecated in
           OTP-23.0 are now removed.

From https://erlang.org/download/OTP-23.0.README

OTP-16232 Application(s): crypto

           As announced in OTP 22.0, a New API was introduced in
           CRYPTO. See the New and Old API chapter in the CRYPTO
           User's Guide for more information and suggested
           replacement functions.

           The Old API is now deprecated in OTP-23.0 and will be
           removed in OTP-24.0.

           This deprecation includes cipher names. See the section
           Retired cipher names in the crypto User's Guide,
           chapter The Old API.

For non cached zones if a zone transfer was interrupted part of the zone is cached in the hot cache.

How can I get ioc2rpz to trust the internally signed certificate? The root CA was added to the host cert store and the root CA is also listed at: /opt/ioc2rpz/cfg/ipa_root_ca.crt

Nov 14 13:05:03 rpz.domain.com de8ed755c4fe[2981461]: TLS client: In state wait_cert_cr at ssl_handshake.erl:2138 generated CLIENT ALERT: Fatal - Unknown CA
Nov 14 13:05:03 rpz.domain.com de8ed755c4fe[2981461]: Error downloading feed <<"https://feed.domain.com/tech/blacklists/block.domains.fqdnlist">> reason {failed_connect,
Nov 14 13:05:03 rpz.domain.com de8ed755c4fe[2981461]:                                                                                                                   [{to_address,
Nov 14 13:05:03 rpz.domain.com de8ed755c4fe[2981461]:                                                                                                                     {"feed.domain.com",
Nov 14 13:05:03 rpz.domain.com de8ed755c4fe[2981461]:                                                                                                                      443}},
Nov 14 13:05:03 rpz.domain.com de8ed755c4fe[2981461]:                                                                                                                    {inet,
Nov 14 13:05:03 rpz.domain.com de8ed755c4fe[2981461]:                                                                                                                     [inet],
Nov 14 13:05:03 rpz.domain.com de8ed755c4fe[2981461]:                                                                                                                     {tls_alert,
Nov 14 13:05:03 rpz.domain.com de8ed755c4fe[2981461]:                                                                                                                      {unknown_ca,
Nov 14 13:05:03 rpz.domain.com de8ed755c4fe[2981461]:                                                                                                                       "TLS client: In state wait_cert_cr at ssl_handshake.erl:2138 generated CLIENT ALERT: Fatal - Unknown CA\n"}}}]}

Rules are incorrectly generated for IPv4 Networks it RPZ type MIXED