mushorg / tanner Goto Github PK

This is a bit of a tricky one and we made various attempts in Glastopf.
First attempt was based on tokenizing know SQL injection queries and using them to match queries against Glastopf. This had limited success as we had a small database. Response was generally just a default MySQL error message.
Second attempt can be found here and here (IIRC libinjection has now it's own Python bindings).
Finally we have this fork of Glastopf using a sandboxed database: https://github.com/rebeccan/glastopf
I think an initial implementation should be based on libinjection.

We should add a SQLi section here: https://github.com/mushorg/tanner/blob/master/docs/source/emulators.rst

Add TANNER to Travis so we get continuous testing.

Input should be a sensor UUID (part of every request to tanner). Something like /api/<uuid>/stats
Should show some basic stats regarding the activity from the sensor.

#103

When TANNER determine the type of vulnerability, it incorrectly handles xss with links (it marks them as rfi).
For example, these links:

'/default.ph/?pageweb=<LINK+REL="stylesheet"+HREF="http://ha.ckers.org/xss.css">'
'/default.ph/?pageweb=<STYLE>@import\\'http://ha.ckers.org/xss.css\\';</STYLE>'
'/default.ph/?pageweb=Redirect+302+/a.jpg+http://victimsite.com/admin.asp&deleteuser'
'/default.ph/?pageweb=<IMG+SRC="http://www.thesiteyouareon.com/somecommand.php?somevariables=maliciouscode">'
'/default.ph/?pageweb=<A+HREF="javascript:document.location=\\'http://www.google.com/\\'">XSS</A>'
'/default.ph/?pageweb=<A+HREF="http://66.102.7.147/">XSS</A>'
'/default.ph/?pageweb=<IFRAME+SRC=http://ha.ckers.org/scriptlet.html+<'
'/default.ph/?pageweb=<META+HTTP-EQUIV="refresh"+CONTENT="0;+URL=http://;URL=javascript:alert(\\'XSS\\');">'
'/default.ph/?pageweb=<XML+SRC="http://ha.ckers.org/xsstest.xml"+ID=I></XML>+<SPAN+DATASRC=#I+DATAFLD=C+DATAFORMATAS=HTML></SPAN>'

We should handle cases like #24 and #25

Store the file with the file name being a md5 hash of the file.
Think about if we want to pass the path to the file or the file content to PHPOX.

We need to validate config before using its values

Related to #5 Create some rules to detect a remote file inclusion.

i have installed the redis-server and php sanbox according to given instructions and so as tanner. redis server and sandbox is working well. but the tanner is not working ...here is the output after i give command sudo tanner in tanner directory.

With some basic test to ensure functionality we are able to see quickly if a pull request breaks the tanner. Probably best to use nosetests and unittest.

By using PHPox as a service we should be able to run injected PHP code and return the results to the adversary in a timely manner. We are primarily interested in executing those PHP scripts that try to evaluate the vulnerability of the web application. Bot's are interesting too but they usually require longer execution time (this could be done separately).

How should we organize logging system? What info should we write to logs? Should we replace print messages with logs?

Instead of using a hash of user agent and IP (

$ sudo tanner
Traceback (most recent call last):
File "/usr/local/bin/tanner", line 4, in
import('pkg_resources').run_script('Tanner==0.1.0', 'tanner')
File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 744, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 1499, in run_script
exec(code, namespace, namespace)
File "/usr/local/lib/python3.5/dist-packages/Tanner-0.1.0-py3.5.egg/EGG-INFO/scripts/tanner", line 21, in
main()
File "/usr/local/lib/python3.5/dist-packages/Tanner-0.1.0-py3.5.egg/EGG-INFO/scripts/tanner", line 16, in main
config.TannerConfig.set_config(args.config)
File "/usr/local/lib/python3.5/dist-packages/Tanner-0.1.0-py3.5.egg/tanner/config.py", line 12, in set_config
TannerConfig.create_default_config(cfg, config_path)
File "/usr/local/lib/python3.5/dist-packages/Tanner-0.1.0-py3.5.egg/tanner/config.py", line 23, in create_default_config
with open(config_path, 'w') as configfile:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/tanner/config.cfg'

Have a look at http://user-mode-linux.sourceforge.net/old/hppfs.html

With session tracking #7 we can:

• Evaluate how successful our responses are in triggering multiple stages
• Attempt fingerprinting the session owner (bots/tools/persons)

This lookup can cause a KeyError:

For now LFI adds files to virtualdocs folder only if the linux directory empty. I think it will be great if we will add new files when new object is put to vdocs.json

Hi, can you update tanner run instructions in the Readme file?. Thanks

How many requests can we serve per second? Which APIs are slow? Which handlers are slow? Does https://github.com/MagicStack/uvloop speed up things?

This line failes if the IP in None:

I think this project has not been selected as a part of GSoC'16.

Will you be working on this in your personal time?

If not, then I'd like to ask you the same thing I've asked before, Can I port this (& snare) to flask? 🙏
I have no technical reason to do this. Just that I'm really well versed with flask (& things like how its projects should be structure etc.)

I actually want to work on this idea in any free time I can find, as I think there's a lot I could learn. But, the project involves a lot of things that are completely foreign to me, and I don't want the choice of a web framework etc. weighing me down (asyncio just feels like one more thing I'll have to learn.) I am just trying to move to towards a comfortable territory so I can focus on stuff that actually matters - the honeynet, vulnerabilities etc.

Let me know what you feel.

I've cloned wordpress site which contains some .css files in the /wp-content structure.
When hosting the clone with SNARE a request to such a css leads to a 'FileNotFound' Error in Tanner and a response 500 Internal Server error on SNARE.

The tanner log contains following error log:

2017-02-08 15:20 INFO:tanner.server.HttpRequestHandler:handle_event: Requested path /wp-content/themes/quark/css/normalize.css?ver=3.0.2
2017-02-08 15:20 ERROR:tanner.server.HttpRequestHandler:log_exception: Error handling request
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/aiohttp/server.py", line 261, in start
    yield from self.handle_request(message, payload)
  File "/usr/lib/python3.6/site-packages/Tanner-0.1.0-py3.6.egg/tanner/server.py", line 92, in handle_request
    response_msg = yield from self.handle_event(data, self.redis_client)
  File "/usr/lib/python3.6/site-packages/Tanner-0.1.0-py3.6.egg/tanner/server.py", line 56, in handle_event
    detection = yield from self.base_handler.handle(data, session, path)
  File "/usr/lib/python3.6/site-packages/Tanner-0.1.0-py3.6.egg/tanner/emulators/base.py", line 74, in handle
    detection = yield from self.emulate(data, session, path)
  File "/usr/lib/python3.6/site-packages/Tanner-0.1.0-py3.6.egg/tanner/emulators/base.py", line 64, in emulate
    detection = yield from self.handle_get(path)
  File "/usr/lib/python3.6/site-packages/Tanner-0.1.0-py3.6.egg/tanner/emulators/base.py", line 53, in handle_get
    sqli = yield from self.emulators['sqli'].check_get_data(path)
  File "/usr/lib/python3.6/site-packages/Tanner-0.1.0-py3.6.egg/tanner/emulators/sqli.py", line 61, in check_get_data
    sqli = yield from self.check_sqli(query[1])
  File "/usr/lib/python3.6/site-packages/Tanner-0.1.0-py3.6.egg/tanner/emulators/sqli.py", line 39, in check_sqli
    res = yield from _run_cmd(command)
  File "/usr/lib/python3.6/site-packages/Tanner-0.1.0-py3.6.egg/tanner/emulators/sqli.py", line 33, in _run_cmd
    proc = yield from asyncio.wait_for(asyncio.create_subprocess_exec(*cmd, stdout=PIPE), 5)
  File "/usr/lib/python3.6/asyncio/tasks.py", line 352, in wait_for
    return fut.result()
  File "/usr/lib/python3.6/asyncio/subprocess.py", line 212, in create_subprocess_exec
    stderr=stderr, **kwds)
  File "uvloop/loop.pyx", line 2164, in __subprocess_run (uvloop/loop.c:39415)
  File "uvloop/handles/process.pyx", line 549, in uvloop.loop.UVProcessTransport.new (uvloop/loop.c:93764)
  File "uvloop/handles/process.pyx", line 87, in uvloop.loop.UVProcess._init (uvloop/loop.c:85151)
FileNotFoundError: [Errno 2] No such file or directory
2017-02-08 15:20 INFO:aiohttp.access:log: 127.0.0.1 - - [08/Feb/2017:15:20:49 +0000] "POST /event HTTP/1.1" 500 170 "-" "Python/3.6 aiohttp/1.2.0"

I'm not sure if it's an TANNER issue because of the ?ver=3.0.2 or an aiohttp/server.py issue.
I would fix it and propose a Pull Request, but I'm not sure what exactly causes the issue.

Right now the LFI emulation is very limited we probably want to add a virtual file system similar to what we have for Glastopf while considering issue #1

How do I run tanner?

I need to see the .log file

Where I find it?

thanks

I don't like that all the files into the root of the project. Maybe we should make the project structure better: for example, move emulators into emulators/handlers folder, change naming and so on

We can get differ types of messages from TANNER, so we should check the type and existence of some dict keys to process the page. https://github.com/mushorg/snare/blob/master/snare.py#L187

TANNER should return unified messages to make processing the result of the emulation easier. For example, now payload can be dict or str, so we should make a lot of checks to choose the action. If TANNER returns unified structured messages we will have the possibility to parse it properly. TANNER should have clear, proper api. Also we can return error messages if something goes wrong and make logs (as a option) of these errors on the SNARE side.

Maybe we can make the types of the messages, for example:

• Type 1 - it's a common query, no attack, nothing to do;
• Type 2 - injectable payload (data should be shown on the current page) ;
• Type 3 - payload with new page;
• Type 4 - error.
  An so on.

So, if we get the type 4, we definitely know that we need to get "error" key from the message.

It's a bit dirty description. Any advice and suggestions will be greatly appreciated

All operational data should be in the /opt/tanner directory.

when i run snare with tanner service i have the follow error:

this is a bug? or why happen?

Thanks!

I'm testing SNARE/TANNER with grabber.
grabber --spider 1 --include --url <snare host url> ,
and OWASP Zap. LFI testing going well, but when starting testing RFI, it fails with this error:

Traceback (most recent call last):
  File "server.py", line 128, in <module>
    loop.run_forever()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 331, in run_forever
    self._run_once()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1262, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/lib/python3.5/selectors.py", line 441, in select
    fd_event_list = self._epoll.poll(timeout, max_ev)
OverflowError: timeout is too large
Task was destroyed but it is pending!
task: <Task pending coro=<ServerHttpProtocol.start() running at /usr/local/lib/python3.5/dist-packages/aiohttp-0.22.0a0-py3.5.egg/aiohttp/server.py:266> wait_for=<Future pending cb=[BaseSelectorEventLoop._sock_connect_done(10)(), Task._wakeup()]>>
Exception ignored in: <generator object ServerHttpProtocol.start at 0x7ff2725cbbf8>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/aiohttp-0.22.0a0-py3.5.egg/aiohttp/server.py", line 312, in start
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 568, in close
  File "/usr/lib/python3.5/asyncio/base_events.py", line 483, in call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 492, in _call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 320, in _check_closed
RuntimeError: Event loop is closed

Currently, Tanner just responds to events by detecting some known patterns, and responding with payloads (if they exist.)

Do we also plan to store these events (in some database perhaps) for analysis later?

I'm also just guessing that this will be done on Tanner's side, rather than Snare's.

I'd be interested in working on this feature (if it is needed, of course,)

P.S: It'd be nice if you could maybe just add some issues about functionalities that you feel makes sense in Snare/Tanner.

It seems that new version of aiohttp fails when trying to connect over ftp.

tanner.rfi_emulator.RfiEmulator: ERROR: Error during downloading the rfi script [Errno 101] Cannot connect to host mirror.yandex.ru:None ssl:False [Can not connect to mirror.yandex.ru:None [Network is unreachable]]

We can use another library for ftp file downloading or try to solve the problem with aiohttp (if it's possible)

Add the capability to track client sessions. Be it through the IP address, user agent, cookies... We had something along those lines in Glastopf.

Now we have some hard coded values: handler's directories, db name, redis host/port. I think it will be great, if we create config file for these purposes.

Right now we have a fixed collection of dorks: https://github.com/mushorg/tanner/blob/master/dorks.pickle
We want to increase the collection of dorks to make the SNARE sensors more attractive.

• Extract dork from request path
• Add new dorks to a separate pickle
• Mix dorks from the original pickle with dorks from the new pickle when requested by SNARE

Need to write tests for config class to avoid problems like this #104 (Bad using of StringIO)

I have cloned site, which has Cyrillic symbols in paths. And I got this error:

--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.6/logging/__init__.py", line 989, in emit
    stream.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 112-115: ordinal not in range(128)
Call stack:
  File "/opt/tanner/bin/tanner", line 33, in <module>
    main()
  File "/opt/tanner/bin/tanner", line 29, in main
    tanner.server.run_server()
  File "/opt/tanner/tanner/server.py", line 132, in run_server
    loop.run_forever()
  File "/usr/lib/python3.6/site-packages/aiohttp/server.py", line 261, in start
    yield from self.handle_request(message, payload)
  File "/opt/tanner/tanner/server.py", line 89, in handle_request
    response_msg = yield from self.handle_event(data, self.redis_client)
  File "/opt/tanner/tanner/server.py", line 54, in handle_event
    self.logger.info('Requested path %s', path)
Message: 'Requested path %s'
Arguments: ('/sites/default/files/images/фото яковец.jpg',)

With the capability of tracking a session #7 can we return injected malicious code to the attacker?

POST /posts?comment=<script>alert('xss');</script>
GET /posts contains the JS

We should have two log files: one for the events and one for the errors

FAIL: test_ftp_download (tanner.tests.test_rfi_emulation.TestRfiEmulator)

Traceback (most recent call last):
  File "/home/travis/build/mushorg/tanner/tanner/tests/test_rfi_emulation.py", line 26, in test_ftp_download
    self.assertIsNotNone(data)
AssertionError: unexpectedly None

tanner.rfi_emulator.RfiEmulator: ERROR: Error during downloading the rfi script [Errno 101] Cannot connect to host mirror.yandex.ru:None ssl:False [Can not connect to mirror.yandex.ru:None [Network is unreachable]]