nbs-system / nxtool-ng Goto Github PK

nxtool should be able to generate type filters.

This is the bug:

root@machine:/var/opt/nxtool-ng# python nxtool.py --whitelist --flat-file /var/log/nginx/site/error.log 

[+] ['2017/06/16 11:43:01 [error] 8090#0: *717302 access forbidden by rule, client: 1.2.3.4, server: site, request: "GET /spip.php?page=login&url=ecrire%2F HTTP/1.1", host: "site"\n is an invalid extlog or nxlog, string "ip=" not found.'] while parsing 2017/06/16 11:43:01 [error] 8090#0: *717302 access forbidden by rule, client: 1.2.3.4, server: site, request: "GET /spip.php?page=login&url=ecrire%2F HTTP/1.1", host: "site"

[+] ['2017/06/16 12:10:04 [error] 8765#0: *717762 access forbidden by rule, client: 5.6.7.8, server: site, request: "GET /spip.php?page=login&url=ecrire%2F HTTP/1.1", host: "site"\n is an invalid extlog or nxlog, string "ip=" not found.'] while parsing 2017/06/16 12:10:04 [error] 8765#0: *717762 access forbidden by rule, client: 5.6.7.8, server: site, request: "GET /spip.php?page=login&url=ecrire%2F HTTP/1.1", host: "site"

[+] ['2017/06/16 12:10:15 [error] 8764#0: *717763 access forbidden by rule, client: 5.6.7.8, server: site, request: "GET /ecrire/ HTTP/1.1", host: "site"\n is an invalid extlog or nxlog, string "ip=" not found.'] while parsing 2017/06/16 12:10:15 [error] 8764#0: *717763 access forbidden by rule, client: 5.6.7.8, server: site, request: "GET /ecrire/ HTTP/1.1", host: "site"

[+] ['2017/06/16 12:14:32 [error] 8798#0: *718044 access forbidden by rule, client: 5.6.7.8, server: site, request: "GET /spip.php?page=login&url=ecrire%2F HTTP/1.1", host: "site"\n is an invalid extlog or nxlog, string "ip=" not found.'] while parsing 2017/06/16 12:14:32 [error] 8798#0: *718044 access forbidden by rule, client: 5.6.7.8, server: site, request: "GET /spip.php?page=login&url=ecrire%2F HTTP/1.1", host: "site"

[+] Generating Google analytics rules
[+] Generating Image 1002 rules
[+] Generating array-like variable name rules
Traceback (most recent call last):
  File "nxtool.py", line 145, in <module>
    sys.exit(main())
  File "nxtool.py", line 130, in main
    rules = module.generate_whitelist(source, whitelist)
  File "/var/opt/nxtool-ng/nxtool/whitelists_generators/__init__.py", line 4, in wrapper
    return func(provider, wl)
  File "/var/opt/nxtool-ng/nxtool/whitelists_generators/array_like_variables_names.py", line 35, in generate_whitelist
    variables = provider.get_top('var_name')
  File "/var/opt/nxtool-ng/nxtool/log_providers/flat_file.py", line 63, in get_top
    for key, value in collections.Counter(values).most_common(10):
  File "/usr/lib/python2.7/collections.py", line 453, in __init__
    self.update(iterable, **kwds)
  File "/usr/lib/python2.7/collections.py", line 534, in update
    for elem in iterable:
  File "/var/opt/nxtool-ng/nxtool/log_providers/flat_file.py", line 62, in <genexpr>
    values = (log[field] for log in self.__get_filtered_logs())
KeyError: 'var_name'

error.txt

Currently,m nxtool is able to generate rules like this one: BasicRule wl:1009,1302,1303 "mz:$ARGS_VAR:ope|$URL:/StoreLocator/" "msg:Variable zone-wide on a specific url".

It would be great to have things like "msg:chars < and > allowed in variable 'ope' on url '/StoreLocator/'" instead of "msg:Variable zone-wide on a specific url".

It would be great to be able to factorize results like those ones:

BasicRule wl:1015 "mz:$URL:/api/fr/contents|$ARGS_VAR:type" "msg:Variable zone-wide on a specific url if it matches a comma";
BasicRule wl:1015 "mz:$URL:/api/ar/contents|$ARGS_VAR:type" "msg:Variable zone-wide on a specific url if it matches a comma";
BasicRule wl:1015 "mz:$URL:/api/en/contents|$ARGS_VAR:type" "msg:Variable zone-wide on a specific url if it matches a comma";

I can't seem to get it working with a flat file even with the example one.

[root@li1305-120 nxtool-master]# python nxtool.py -v --flat-file --whitelist
INFO:root:Running Google analytics
INFO:root:Running Image 1002
INFO:root:Running cookies
INFO:root:Running url_wide_id
INFO:root:Searching for aguments in the zone ARGS
Traceback (most recent call last):
  File "nxtool.py", line 99, in <module>
    sys.exit(main())
  File "nxtool.py", line 92, in main
    whitelist.extend(module.generate_whitelist(source, whitelist))
  File "/home/nxtool-master/nxtool/whitelists_generators/__init__.py", line 4, in wrapper
    return func(provider, wl)
  File "/home/nxtool-master/nxtool/whitelists_generators/zone_var_wide.py", line 19, in generate_whitelist
    search = provider.export_search()
AttributeError: 'FlatFile' object has no attribute 'export_search'
[root@li1305-120 nxtool-master]# 

--stdin option seems broken it might be worth doing something even if get rid of this option.

nxtool-ng just generated the following rules,something is wrong:

Generated whitelists:
	BasicRule wl:1013 "mz:$HEADERS_VAR:cookie" "msg:Cookies that matches a simple quote";
	BasicRule wl:1013 "mz:HEADERS" "msg:zone-wide ID whitelist if it matches a simple quote";
	BasicRule wl:1013 "mz:HEADERS" "msg:Site-wide id+zone if it matches simple quote";

Hello,

I run this tool against a test error log and it returned this message

Command: python nxtool.py --whitelist --flat-file=/var/www/error.log

Result: nxtool was not able to generate meaningful whitelist

When I run the same test error log against the nx_util.py included int Naxsi version 0.53.2 it does generate this white list rules.

########### Optimized Rules Suggestion ##################
# total_count:2 (20.0%), peer_count:1 (50.0%) | sql keywords
BasicRule wl:1000 "mz:$URL:/naxsi2/wp-includes/js/imgareaselect/imgareaselect.css|URL";
# total_count:1 (10.0%), peer_count:1 (50.0%) | close square bracket (]), possible js
BasicRule wl:1311 "mz:$URL:/naxsi2/|$BODY_VAR:ips[0]|NAME";
# total_count:1 (10.0%), peer_count:1 (50.0%) | open square backet ([), possible js
BasicRule wl:1310 "mz:$URL:/naxsi2/|$BODY_VAR:ips[0]|NAME";
# total_count:1 (10.0%), peer_count:1 (50.0%) | double encoding
BasicRule wl:1315 "mz:$URL:/naxsi2/|$HEADERS_VAR:cookie";

Any idea why nxtool is not creating these rules?

Hi,

I've copied naxsi-0.55.1/nxapi/nxapi into /usr/lib/python2.7/site-packages/ but when doing "python nxtool.py -h", I get:

<<
Traceback (most recent call last):
  File "nxtool.py", line 5, in <module>
    from nxapi import whitelist as nxapi_whitelist
ImportError: cannot import name whitelist
>>

Where can I get this "whitelist" ?

Thx !

Some of the whitelist generator ask for a lot of samples before returning a whitelist. This configuration is hardcoded. It seems to be a bit confusing since we have that --slack option. We may at least loosen constraints when slack is activated.

Hi I am attempting to get our logs into an ElasticSearch V5 server but am getting the following error. could you help ?

nxtool-ng-master]# python ./nxtool.py --flat-file /root/Z.log --elastic-dest
Traceback (most recent call last):
  File "./nxtool.py", line 153, in <module>
    sys.exit(main())
  File "./nxtool.py", line 129, in main
    destination.insert([log])
  File "/root/nxtool-ng-master/nxtool/log_providers/__init__.py", line 53, in insert
    self.nList.extend(obj)
AttributeError: 'Elastic' object has no attribute 'nList'

Thanks
Keith

It would be great to have regressive matching on url, like:

• /pif/paf/pouf/test.gif
• /pif/paf/pouet
• /pif/pof

would result in a rule on ^/pif/.

Hello,

There are a few glitches in the whitelists generated by nxtool.
At the moment, I noticed two of them :

1

You can't mix BODY_VAR and ARGS_VAR in the same rule:

BasicRule wl:1310,1311 "mz:$BODY_VAR_X:data\[.+\]|$ARGS_VAR_X:data\[.+\]" "msg:Array-like variable name";

You should use to rules, on with "mz:$URL:/url|$BODY_VAR:data"; and the other with s/BODY/ARGS/

2

The args or body var name whitelist is incorrectly set:

BasicRule wl:1000 "mz:ARGS|NAME:yes" "msg:Variable zone-wide";

Should be :

BasicRule wl:1000 "mz:$ARGS_VAR:yes|NAME" "msg:Variable zone-wide";

Please add a --tag option in order to tag elements as whitelisted in an ElasticSearch db.

nxtool should support filters, like Please give me everything that triggered an alert in the URL.

The internal details of the ES data strucutre used in nxtool-ng should be documented.

Currently filters in nxtool are static. A filter for var_name: g-recaptcha-response can easily be added whereas a filter for comments: import:2016-09-29 08:22:17.215923,Whitelisted:2016-09-30 10:09:31.724727 cannot be added since the last part is modified for each event.

Could you please add a regex support for the --filter option. This way users could filter fields like this --filter "comments:.*Whitelisted.*"

Hi, If I try and import / parse ? a log file with nxtool-ng it fails with the following error. I am running the command on a new Centos 7.3 VM. Any help would be appreciated.

# python ./nxtool-ng-master/nxtool.py --flat-file /root/Y.log
Traceback (most recent call last):
  File "./nxtool-ng-master/nxtool.py", line 153, in <module>
    sys.exit(main())
  File "./nxtool-ng-master/nxtool.py", line 149, in main
    print(printers.print_generic(source.get_results()))
  File "/root/nxtool-ng-master/nxtool/printers.py", line 18, in print_generic
    print('\n'.join('%s: %s' % (k, item[k]) for k in item) + '\n')
  File "/root/nxtool-ng-master/nxtool/printers.py", line 18, in <genexpr>
    print('\n'.join('%s: %s' % (k, item[k]) for k in item) + '\n')
TypeError: list indices must be integers, not dict

Thanks
Keith.

The current implementation doesn't work, fix it.

Don't generate too wide whitelists :

• when generating url-wide WL(s), check that at least X different URLs are impacted
  etc :)
• when generating zone-wide WL(s), check that at least X different named arguments are impacted

--typing seems broken

Find a way to ensure that a logline is imported only once.

It would be cool if nxtool could import naxsi logs from

• archives
  • zip
  • tar
• stdin
• elastic search
• flat files

Hello there,
if I run nxtool.py with --whitelist --flatfile /var/log/nginx/error.log and the log has stuff like
[alert] 16676#16676: worker process 16678 exited on signal 11 (core dumped)\n is an invalid extlog or nxlog, string "ip=" not found.'] while parsing 2017/04/10 14:43:27 [alert] 16676#16676: worker process 16678 exited on signal 11 (core dumped) the whitelist generation fails.

Perhaps ignore stuff like that?

It would be great to be able to generate useless fancy statistics with nxtool

Given some naxsi logs, nxtool should be able to generate whitelists, for example:

• cookies is a specific case of WL generation
• some variables have [ and ] in their names (eg array[1]=2&array[2]=1337)

Hello again, I try running python nxtool.py --stats --elastic and the command fails with

elasticsearch.exceptions.RequestError: TransportError(400, u'illegal_argument_exception', u'No search type for [count]')

This works using the original nxtool, elasticsearch is 5.3.0 on a ubuntu 16.04 system.

jvoisin@mim 18:00 ~/Dev/nxtool python3 ./nxtool.py --elastic --whitelist www.example.com                                                                                                [master] git:nxtool
/home/jvoisin/.local/lib/python3.5/site-packages/elasticsearch/connection/http_urllib3.py:70: UserWarning: Connecting to 10.0.9.25 using SSL with verify_certs=False is insecure.
  'Connecting to %s using SSL with verify_certs=False is insecure.' % host)
[+] Generating Google analytics rules
[+] Generating Image 1002 rules
[+] Generating array-like variable name rules
[+] Generating cookies rules
[+] Generating var + zone rules
[+] Generating url rules
[+] Generating var + zone rules
[+] Generating zone rules
[+] Generating site rules

Generated whitelists:
	BasicRule wl:1015 "mz:$URL:/poll/8|$BODY_VAR:ajax_page_state[libraries]" "msg:Variable zone-wide on a specific url";
	BasicRule wl:1001,1311,1310,1303 "mz:$URL:/politique/<img src="https:/s372.example.net/bb-mx/prime|$ARGS_VAR:tm" "msg:Variable zone-wide on a specific url";
	BasicRule wl:1302 "mz:$URL:/<img" "msg:url-wide ID whitelist";
	BasicRule wl:1302 "mz:$URL:/politique/<img src="https:/s372.example.net/bb-mx/prime" "msg:url-wide ID whitelist";
	BasicRule wl:1009,18 "mz:ARGS:" width" "msg:Variable zone-wide";
	BasicRule wl:1011,1010 "mz:ARGS" "msg:zone-wide ID whitelist"

jvoisin@mim 18:00 ~/Dev/nxtool  

The " width variable isn't properly escaped.

It would be great to be able to reduce rules up to var_name, like HEADERS:content-type

Being able to display the current ratio of whitelisted/non-whitelisted events, such as :

nbs-system/naxsi#381

From time to time, nxtool-ng does not find whitelists for a particular vhost, the former version of the nxtool project had a --slack option to provide users with a full list of basic whitelists.

Can you add this feature?

It will be useful if the --flat-file argument can support globing, nxapi will be able to process many files at the same time.

example : python nxtool.py --flat-file /mydirectory/error.log.*