tomchop / malcom Goto Github PK

Malcom - Malware Communications Analyzer

License: Other

Python 75.73% CSS 1.81% JavaScript 6.61% HTML 15.73% Shell 0.12%

dfir infosec malware malware-analysis network-traffic pcap threat-intelligence

malcom's Introduction

Hi there 👋

I'm Tom (he/him), a Digital Forensics and Incident Response (a.k.a. DFIR) engineer based in Zurich, Switzerland. Most of my focus is around tools that aid in incident response, forensics, threat intelligence, malware analysis, automation, and API interaction.

📯 Where to find me

Keybase - https://keybase.io/tomchop
Mastodon - @[email protected]
Twitter - @tomchop_ (don't check this much anymore)
tomchop.me - Personal website, let's see if I ever start writing something there...

⚡️ Core projects

dfTimewolf - a digital forensics pipeline orchestrator. Think CyberChef for APIs! Actively maintained.
Yeti platform - a lightweight Threat Intelligence platform. Ramping up the time I'm spending on this.
Timesketch - a forensics timeline analysis platform.

📦 Projects I've worked on in the past

volatility-autoruns - A plugin for the excellent memory analysis framework Volatility that enumerates auto-start extensibility points (i.e. "persistence") on a system.
FIR - Fast incident response - a lightweight incident response platform. Like a ticketing system, but for security incidents.
unxor - A fun experiment attacking weaknesses in XOR-based ciphers. Allows you to recover plaintext from any fixed-key XOR ciphertext, as long as you know a chunk of plaintext that is 2x as long as the key! (e.g. This program cannot be run in DOS mode)
malcom - Malcom - Malware Communications Analyzer - network traffic analysis and threat intelligence in the browser.

malcom's People

Contributors

Stargazers

Watchers

Forkers

pr0hest idkwim 0day1day ant4g0nist xdancho caliskanfurkan tania188 riyantech darkwarriors vsantola 2xyo zertrin spnow yampolskiy blaquee mikalv neslog the-darknight vicgc synchroack xshield360 ch40s syphersec lazycrazyowl kingtuna cnbird1999 eldraco ababook cyj chubbymaggie loveshell darron debug-orz mowadev01 signedbytes hackmycontrolsystem aburan28 victorfang ksmaheshkumar sebdraven jbertman noconstruct mallocnode kernux securityigi tourountzis valour01 mayur021 sukelluskello arabians lucabongiorni abdullah-mughal jipegit he0x namegpark sh4d0ws4int johnjohnsp1 yd0str melbshark xuanhan863 m4z3n blackosint ww9210 thurday cybertaoflow linkp2p weixu8 binsrc laudarch oped fo0nikens tdr130 johnfellers sbkiller r00tak oi-buhtig msmakhlouf br0k3ngl255 pulkitpagare bikerdroid rajivraj andy737 security-geeks geezer-workshop diggold kumaraguruv openspherelab tun4 mmorenog 10tohh beelives caar2000 cloudxtreme batidiane wxdublin securitywarrior krennos pseudozidu dragxn redteamcaliber

malcom's Issues

API: Retrieve data from sniffing session

Use the API to retreive data from a sniffing session given its ID to:

Retreive all data regarding it (nodes, edges, flows, etc.)
Retreive all elements associated to it (hostnames, ips, urls)
Retreive only the evil elements associated to it

My struggles with malcom

First off Tom this is a great app. I really enjoy it and it's very unique. That being said here are my struggles with it currently.

So, my setup is this is behind a firewall and IDS...and as soon as Malcom starts with feeds my IDS lights up all over the place. So, ultimately, my goal is to have Malcom without the feeds options. Things I've tried:

From source. The first sticking point is cryptography==0.7.2 on debian/ubuntu won't install at all. Removing the line and running the pip command auto installs cryptography-2.0.3. After changing feeds = false here's what I get after the run:

{'MAX_WORKERS': '12', 'SKIP_TAGS': 'whitelisted', 'ACTIVATED': 'true'}
{'WEB': True, 'LISTEN_PORT': 8080, 'ANALYTICS': True, 'ACTIVATED': 'true', 'AUTH': False, 'MAX_WORKERS': 12, 'SKIP_TAGS': ['whitelisted'], 'VERSION': '1.3a', 'LISTEN_INTERFACE': '0.0.0.0'}
Detected interfaces:
WARNING: Failed to execute tcpdump. Check it is installed and in the PATH
WARNING: No route found for IPv6 destination :: (no default route?)
[+] Starting sniffer...
[+] Successfully loaded sniffer directory: /opt/malcom/Malcom/sniffer/captures
Traceback (most recent call last):
  File "./malcom.py", line 79, in <module>
    setup.sniffer_engine = netsniffer.SnifferEngine(setup)
  File "/opt/malcom/Malcom/sniffer/netsniffer.py", line 41, in __init__
    from Malcom.sniffer.tlsproxy.tlsproxy import MalcomTLSProxy
  File "/opt/malcom/Malcom/sniffer/tlsproxy/tlsproxy.py", line 9, in <module>
    from twisted.internet import defer, ssl
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/twisted/internet/ssl.py", line 59, in <module>
    from OpenSSL import SSL
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 11, in <module>
    from OpenSSL._util import (
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 3, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 13, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
AttributeError: 'module' object has no attribute '_init_cffi_1_0_external_module'

after this you can't even run pip anymore:

Traceback (most recent call last):
  File "/opt/malcom/env-malcom/bin/pip", line 7, in <module>
    from pip import main
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/pip/__init__.py", line 21, in <module>
    from pip._vendor.requests.packages.urllib3.exceptions import DependencyWarning
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/pip/_vendor/__init__.py", line 64, in <module>
    vendored("cachecontrol")
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/pip/_vendor/__init__.py", line 36, in vendored
    __import__(modulename, globals(), locals(), level=0)
  File "/opt/malcom/env-malcom/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/__init__.py", line 9, in <module>
  File "/opt/malcom/env-malcom/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/wrapper.py", line 1, in <module>
  File "/opt/malcom/env-malcom/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/adapter.py", line 4, in <module>
  File "/opt/malcom/env-malcom/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/__init__.py", line 52, in <module>
  File "/opt/malcom/env-malcom/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/packages/__init__.py", line 59, in <module>
  File "/opt/malcom/env-malcom/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/packages/__init__.py", line 32, in vendored
  File "/opt/malcom/env-malcom/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/contrib/pyopenssl.py", line 47, in <module>
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 11, in <module>
    from OpenSSL._util import (
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 3, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 13, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
AttributeError: 'module' object has no attribute '_init_cffi_1_0_external_module'

the entire virtual-env seems destroyed.

Docker. So the docker image works fine, however, again, as soon as it's fired up it sprays all over the IDS causing a lot of issues. I've tried: routing all the traffic over tor, blocking dns. What I'd LIKE to do is modify the malcom.conf file within the but I haven't been successful.

Tom, is there any way you can add a feature in the web page configuration to disable feeds? Just...SOMETHING that will stop the feeds. Thank you

cryptography

have encountered problem during install!
ommand "/root/malcom/env-malcom/bin/python2 -u -c "import setuptools, tokenize;file='/tmp/pip-build-PX89Xx/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-tybGJc-record/install-record.txt --single-version-externally-managed --compile --install-headers /root/malcom/env-malcom/include/site/python2.7/cryptography" failed with error code 1 in /tmp/pip-build-PX89Xx/cryptography/

System Kali Linux Rolling 2016.2

Add Signature Malware

Hi..
This tools is really owsome, i have an idea, how if you add some tools to get the signature, country name and location for malware analysis

Thanks for you and thanks for malcom

Add tags to sniffing sessions

Sniffing sessions should have editable tags (and maybe even an optional description field).

Browse dataset error: "AttributeError: 'NoneType' object has no attribute 'lower'"

Whenever I visit the "Browse dataset" page, the page shows "Populating table..." without bringing any data and throws the following error:

Traceback (most recent call last):
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/gevent/pywsgi.py", line 508, in handle_one_response
    self.run_application()
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/geventwebsocket/handler.py", line 88, in run_application
    return super(WebSocketHandler, self).run_application()
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/gevent/pywsgi.py", line 494, in run_application
    self.result = self.application(self.environ, self.start_response)
  File "/opt/malcom/Malcom/web/webserver.py", line 647, in malcom_app
    return app(environ, start_response)
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask_restful/__init__.py", line 265, in error_router
    return original_handler(e)
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask_restful/__init__.py", line 262, in error_router
    return self.handle_error(e)
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask_restful/__init__.py", line 265, in error_router
    return original_handler(e)
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask_restful/__init__.py", line 262, in error_router
    return self.handle_error(e)
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask_restful/__init__.py", line 446, in wrapper
    resp = resource(*args, **kwargs)
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask_login.py", line 755, in decorated_view
    return func(*args, **kwargs)
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/views.py", line 84, in view
    return self.dispatch_request(*args, **kwargs)
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask_restful/__init__.py", line 550, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/opt/malcom/Malcom/web/api.py", line 121, in get
    Model.add_to_history(query.get('value'))
  File "/opt/malcom/Malcom/model/model.py", line 414, in add_to_history
    if query.lower().strip() != '':
AttributeError: 'NoneType' object has no attribute 'lower'

KeyError: 'FEEDS_DIR'

I get the following error when running ./malcom.py in env-malcom :

===== Malcom 1.3a - Malware Communications Analyzer =====

Traceback (most recent call last):
File "./malcom.py", line 59, in
setup.load_config(args)
File "/home/erbu/malcom/Malcom/config/malconf.py", line 14, in load_config
self.sanitize_paths()
File "/home/erbu/malcom/Malcom/config/malconf.py", line 22, in sanitize_paths
if not self['FEEDS_DIR'].startswith('/'):
KeyError: 'FEEDS_DIR'

Consider Removing GeoLiteCity.dat

in /Malcom/auxiliary/GeoIP there is an 18mb file GeoLiteCity.dat. For licensing etc, you may want to include explicit instructions on how users can download that file, but not include it in the git. It will also make the package much smaller. Corresponding with that, perhaps if it doesn't exist there, there can be some clear and concise warnings on it should be installed.

Api & account settings not available ?

Hello,

I'm trying 1.2 alpha and it find very light and visual, especially compare to alternatives :)
My current problem, it seems the api links are not available

$ curl http://127.0.0.1:8080/api/query?value=malcom.io
This page does not exist

checking webserver.py
$ curl http://127.0.0.1:8080/public/api?value=malcom.io
[]
$ curl http://127.0.0.1:8080/evil?value=bevrifuli.geohats.com
{"nodes": [], "edges": []}
$ curl http://127.0.0.1:8080/api/analytics
Internal Server Error
= ValueError: View function did not return a response

and grepping account inside source just returns one line
$ grep -rin account *
web/static/jquery/jquery-ui.js:13781: // into account and update option properly.
= No /account/settings to generate API key

unreleased stuff? or I missed a doc/code ?

Thanks Tom

Continuous Analysis of a URL - how to stop/clear it?

I routed traffic from a host through malcom to www.ford.com and now everytime I start it the app keeps analyzing metrics.ford.com for hours and never stops. Any idea what it going on here and how to clear it out? Here is an example of the logs:

[question] Malware communicating using tor network

I have been worrying that some malware use Tor Nodes to communicate towards the C&C. Are you tracing that also? if yes, how?

Any hint or help will be appreciated.

Bug when no HTTP payload is found over port 80

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 238, in run
    self.load_pcap()
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 215, in load_pcap
    self.sniff(stopper=self.stop_sniffing, filter=self.filter, prn=self.handlePacket, stopperTimeout=1, offline=self.engine.setup['SNIFFER_DIR']+"/"+filename)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 658, in sniff
    r = prn(p)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 509, in handlePacket
    self.send_flow_statistics(flow)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 563, in send_flow_statistics
    data['flow'] = flow.get_statistics(self.engine.yara_rules)
  File "/opt/malcom/Malcom/networking/flow.py", line 299, in get_statistics
    self.decoded_flow = Decoder.decode_flow(self)
  File "/opt/malcom/Malcom/networking/flow.py", line 23, in decode_flow
    data = Decoder.HTTP_request(flow.payload)
  File "/opt/malcom/Malcom/networking/flow.py", line 97, in HTTP_request
    host = re.search(r'Host: (?P<host>[.\w-]+)(:(?P<port>[\d]{1,5}))?', payload).groupdict()
AttributeError: 'NoneType' object has no attribute 'groupdict'

Got this error while processing this pcap: http://www.snaketrap.co.uk/pcaps/hbot.pcap
It might not be related to the previous, so feel free to track it as a separate issue.

Max Upload Pcap

Hi..

Can you tell me, what max size pcap that can i analize?
And how to modify script to resize max upload pcap?

Thanks

Stop the uploading of a pcap.

It would be nice to have some way of stopping the uploading of a pcap. Sometimes it is too big and you don't want to wait so long. Or maybe it is just too big.

thanks!

TypeError

In dev (311b1e5)

I should say that I modify the file Malcom/networking/netsniffer.py and changed the line
self.filter = "ip and not host 127.0.0.1 and not host %s %s" % (remote_addr, filter_ifaces)
for
self.filter = ""

So I can sniff my own traffic.

[MODEL] - (updated hostname ip1a-lb3-prd.iad.github.com)
[MODEL] - (updated hostname 1.0.0.127.in-addr.arpa)
[DEBUG] - Caught DNS question: 1.0.0.127.in-addr.arpa
[DEBUG] - [+] DNS replies caught (1 answers)
[DEBUG] - No relevant records in reply
[DEBUG] - [+] DNS replies caught (1 answers)
[DEBUG] - No relevant records in reply
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(_self.__args, *_self.__kwargs)
File "/home/user/dir/aplics/malcom/Malcom/networking/netsniffer.py", line 90, in run
self.pkts += self.sniff(stopper=self.stop_sniffing, filter=self.filter, prn=self.handlePacket, stopperTimeout=1)
File "/home/user/dir/aplics/malcom/Malcom/networking/netsniffer.py", line 513, in sniff
r = prn(p)
File "/home/user/dir/aplics/malcom/Malcom/networking/netsniffer.py", line 382, in handlePacket
new_elts, new_edges = self.checkHTTP(flow)
File "/home/user/dir/aplics/malcom/Malcom/networking/netsniffer.py", line 323, in checkHTTP
if url['value'] not in self.nodes_values:
TypeError: list indices must be integers, not str

Can

Play/Cancel/Start/Stop buttons with --public

Above mentionned buttons are not printed when malcom is launched with --public parameter.

This occurs because of this line:
{% if not g.config['PUBLIC'] %}

in sniffer html templates.

[Honeynet GSoC 2015] Malcom - Malware communications analyzer

Malcom & GSoC 2015

Malcom is a platform that allows to cross-reference network traffic with different malware feeds (or any other source of data)
Malcom is participating as part of the Honeynet Project in the Google Summer of Code 2015. There are a few ideas I think would be interesting to see included in Malcom. Feel free to comment below and suggest your own ideas or improvements!

This issue is for students that would be interested in contributing to Malcom as part of their participation in GSoC, so that they can have any questions on Malcom or GSoC answered easily.

Main project goals

Build additional traffic-analysis features: DNS request timeline, Suricata or Bro alerts, or find some other way to ID the traffic and write specific decoders for malware protocols;
Share Malcom's data and build an web API that can be queried from other services (FIR, CRITS, MISP)
- Secondary objective: adjust the data model so that it always uses the web API, even locally. This would allow for querying remote Malcom instances as if they were local in total transparency.
Less interesting, but still needs to be done: improving the UI, adding details and tags to elements, improve performance, code clean-up, etc. (this is the part I will probably end-up coding if I get help on any of the points above)

FAQ

1- What background is necessary to contribute to this project?

Solid Python skills;
a working knowledge of current network traffic analysis tools and tech (scapy, dpkt, Bro IDS, Suricata, etc.) so as to leverage them in Malcom;
experience with the Flask framework, d3js and mongodb is definitely a plus.

2- How do I get started?

Download Malcom, play around with it. There's a Docker instance to get you started, but you're encouraged to read through the code, too 😉
Feel free to fork and to pull-request
If you feel like Malcom is a project you could and would like to contribute to, submit a project proposal.

Don't hesitate to contact me if you have additional questions (Twitter or email works fine). There's also a Honeynet GSoC mailing list and a malcom-users Google Group. Feel free to ping me whenever you want.

Project proposal

You'll need to write a project proposal before final approval of your participation. This is mainly a document stating your approach to work on one of the points listed in the project goals (or any other ideas you'd like to work on). Apparently, a rough timeline is needed (knowing you will have around 12 weeks to make the magic happen!).

Screenshot are all broken, incl README and Wiki

Screenshots are broken everywhere I've seen.

problem when running malcom

Hi
when I try to run malcom, I get this error message. I used the default configuration file

./malcom.py -c malcom.conf
===== Malcom 1.3a - Malware Communications Analyzer =====

Traceback (most recent call last):
  File "./malcom.py", line 59, in <module>
    setup.load_config(args)
  File "~/Malcom/config/malconf.py", line 13, in load_config
    self.parse_command_line(args)
  File "~/Malcom/config/malconf.py", line 30, in parse_command_line
    self.parse_config_file(args.config)
  File "~/Malcom/config/malconf.py", line 73, in parse_config_file
    self['MODULES_DIR'] = config.get('sniffer', 'modules_dir')
  File "/usr/lib/python2.7/ConfigParser.py", line 618, in get
    raise NoOptionError(option, section)
ConfigParser.NoOptionError: No option 'modules_dir' in section: 'sniffer'

Compile error running: pip install flask pymongo pygeoip gevent-websocket python-dateutil netifaces

i got this error when try to compile and run the last command from the guide:

Running setup.py install for netifaces

Running command /root/tomchop/malcom/env-malcom/bin/python -c "import setuptools;__file__=$
running install
running build
running build_ext
checking for getifaddrs... not found. (cached)

checking for getnameinfo... not found. (cached)

checking for socket IOCTLs... not found. (cached)

checking for optional header files... none found. (cached)

checking whether struct sockaddr has a length field... no. (cached)

checking which sockaddr_xxx structs are defined... none! (cached)

building 'netifaces' extension

gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC $

netifaces.c:1:20: error: Python.h: No such file or directory

netifaces.c:143:6: error: #error You need to add code for your platform.

netifaces.c: In function 'our_getnameinfo':

netifaces.c:200: warning: implicit declaration of function 'sprintf'

netifaces.c:200: warning: incompatible implicit declaration of built-in function 'sprintf'

netifaces.c:203: warning: implicit declaration of function 'strncpy'

netifaces.c:203: warning: incompatible implicit declaration of built-in function 'strncpy'

netifaces.c:225: warning: incompatible implicit declaration of built-in function 'sprintf'

ImportError: No module named flask_restful

I get the following error, do you think something goes wrong with flash? I have Flash 0.10.1 installed.

Traceback (most recent call last):
  File "./malcom.py", line 112, in <module>
    from Malcom.web.webserver import MalcomWeb
  File "/opt/malcom/Malcom/web/webserver.py", line 120, in <module>
    from Malcom.web.api import malcom_api
  File "/opt/malcom/Malcom/web/api.py", line 7, in <module>
    from flask_restful import Resource, reqparse, Api
ImportError: No module named flask_restful

UI: navbar overflows content container on machine with >3 interfaces

On machines with multiple interfaces (>3) the navigation bar overflows the below content container (tabs, graphs, ...).
Maybe add a test that for more than 2 interfaces it should not list them all but provides the interface list in a tooltip?

errors on startup

I installed Malcom as a Docker container in a new, stock Ubuntu 14.04 VM (& also from github, same result). When I try to run it using the syntax from the Docker part of the README.md (docker run -p 8080:8080 -d --name malcom tomchop/malcom-automatic (I also tried tomchop/malcom, same result)) I get this set of errors:

Starting database mongodb
...done.
Starting redis-server: redis-server.
TERM environment variable not set.
===== Malcom 1.3a - Malware Communications Analyzer =====

Detected interfaces:
eth0: 172.17.0.3
WARNING: Failed to execute tcpdump. Check it is installed and in the PATH
WARNING: No route found for IPv6 destination :: (no default route?)
[+] Starting sniffer...
[+] Successfully loaded sniffer directory: /opt/malcom/Malcom/sniffer/captures
[+] Starting TLS proxy on port 9000
Traceback (most recent call last):
File "./malcom.py", line 79, in
setup.sniffer_engine = netsniffer.SnifferEngine(setup)
File "/opt/malcom/Malcom/sniffer/netsniffer.py", line 51, in init
self.model = Model(self.setup)
File "/opt/malcom/Malcom/model/model.py", line 46, in init
read_preference=read_pref[db_setup.get('READ_PREF', 'PRIMARY')])
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 377, in init
raise ConnectionFailure(str(e))
pymongo.errors.ConnectionFailure: [Errno 111] Connection refused

FT, Hide nodes in graph view based on filter

It will be also nice, if it could be possible to hide things based on the filter selection.
For example to quickly remove all google stuff, a things like "not google" which hide google ip's and hosts

Overflow Error while loading a 3.6GB pcap file

This happen on master (057f471) and dev (311b1e5) branches.

[DEBUG] - [-] No TLS interception
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line 508, in handle_one_response
self.run_application()
File "/usr/local/lib/python2.7/dist-packages/geventwebsocket/handler.py", line 84, in run_application
return super(WebSocketHandler, self).run_application()
File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line 494, in run_application
self.result = self.application(self.environ, self.start_response)
File "/home/user/dir/aplics/malcom/Malcom/web/webserver.py", line 76, in malcom_app
return app(environ, start_response)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1836, in call
return self.wsgi_app(environ, start_response)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/user/dir/aplics/malcom/Malcom/web/webserver.py", line 401, in sniffer
Malcom.sniffer_sessions[session_name].pcap = file.read()
OverflowError: requested number of bytes is more than a Python string can hold
{'CONTENT_LENGTH': '3884500716',
'CONTENT_TYPE': 'multipart/form-data; boundary=----WebKitFormBoundaryDuuQMOS7h3fOpAHu',
'GATEWAY_INTERFACE': 'CGI/1.1',
'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8',
'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch',
'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.8,es;q=0.6',
'HTTP_CACHE_CONTROL': 'max-age=0',
'HTTP_CONNECTION': 'keep-alive',
'HTTP_HOST': '192.168.1.20:8080',
'HTTP_ORIGIN': 'http://192.168.1.20:8080',
'HTTP_REFERER': 'http://192.168.1.20:8080/sniffer/',
'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36',
'PATH_INFO': '/sniffer/',
'QUERY_STRING': '',
'REMOTE_ADDR': '192.168.1.20',
'REMOTE_PORT': '48701',
'REQUEST_METHOD': 'POST',
'SCRIPT_NAME': '',
'SERVER_NAME': 'localhost',
'SERVER_PORT': '8080',
'SERVER_PROTOCOL': 'HTTP/1.1',
'SERVER_SOFTWARE': 'gevent/1.0 Python/2.7',
'werkzeug.request': <Request 'http://192.168.1.20:8080/sniffer/' [POST]>,
'wsgi.errors': <open file '', mode 'w' at 0xb748c0d0>,
'wsgi.input': <gevent.pywsgi.Input object at 0xa7ed3fac>,
'wsgi.multiprocess': False,
'wsgi.multithread': False,
'wsgi.run_once': False,
'wsgi.url_scheme': 'http',
'wsgi.version': (1, 0)} failed with OverflowError

ARP support

Inclusion of ARP requests / replies in network capture

IPv6 support

IPv6 support in analytics and sniffer mode

Errors - Ubuntu 14.04

I am on Ubuntu server 14.04 LTS VM with python 2.7.6 installed. I needed to install libssl-dev and libffi-dev using apt-get in order pyopenssl and all the rest to be installed without errors. I also installed service_identity using pip to avoid a warning when running malcom. I think I don't forget something else... :)

I guess something is still missing because I get the following when I run malcom:

./malcom.py -a
....
[DEBUG] - Could not send message: 'NoneType' object has no attribute 'send'

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

./malcom.py -f
...
[DEBUG] - Starting thread for feed TorExitNodes...
[DEBUG] - Could not send message: 'NoneType' object has no attribute 'send'
Exception in thread Thread-5:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(_self.__args, *_self.__kwargs)
File "/opt/malcom/Malcom/feeds/feed.py", line 62, in run
status = self.update()
File "/opt/malcom/Malcom/feeds/dshield_as16276.py", line 23, in update
self.analyze(line)
File "/opt/malcom/Malcom/feeds/dshield_as16276.py", line 40, in analyze
ip, status = self.analytics.save_element(ip, with_status=True)
File "/opt/malcom/Malcom/analytics/analytics.py", line 81, in save_element
return self.data.save(element, with_status=with_status)
File "/opt/malcom/Malcom/model/model.py", line 123, in save
status = self.elements.update({'value': element['value']}, {"$set" : element, "$addToSet": {'tags' : {'$each': tags}}}, upsert=True)
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/pymongo/collection.py", line 561, in update
check_keys, self.uuid_subtype), safe)
InvalidDocument: Cannot encode object: {'refresh_period': 259200, 'type': 'ip', 'value': '178.32.230.51'}

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
When trying to delete a session:

Traceback (most recent call last):
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/gevent/pywsgi.py", line 508, in handle_one_response
self.run_application()
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/geventwebsocket/handler.py", line 88, in run_application
return super(WebSocketHandler, self).run_application()
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/gevent/pywsgi.py", line 494, in run_application
self.result = self.application(self.environ, self.start_response)
File "/opt/malcom/Malcom/web/webserver.py", line 76, in malcom_app
return app(environ, start_response)
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1836, in call
return self.wsgi_app(environ, start_response)
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/opt/malcom/Malcom/web/webserver.py", line 448, in sniffer_session_delete
g.a.data.del_sniffer_session(session_name)
File "/opt/malcom/Malcom/model/model.py", line 99, in del_sniffer_session
filename = session['name'] + ".pcap"
TypeError: 'NoneType' object has no attribute 'getitem'

Continuous analysis of heartbeat

Since I started the tool, it is continually indicating by command line the analysis of heartbeat is it normal?

Thanks :)

pcapng support?

Gave a try to honeynet14 pcapng but it doesn't seem to load. I suppose pcapng is not supported ?

Change the way elements are associated to a sniffing session

Elements are associated to a sniffing session via tags: the name of a sniffing session will be added as a tag to the element. This is not flexible.

Enhancement: add a new field to the element containing an array to all sniffing session IDs it is present in. When a sniffing session is created and an element is added to the database from this sniffing session, it will add its own ID to the array.

Database configuration

Hi,

I have just downloaded malcom to test it but I am unable to launch it simply because:

my database is on another computer than the one I want to run malcom on
I have configured authentication on my MongoDB

I see you have created a db_local section in the malcom.conf file but, as my comprehension of the code goes, it does not seem to be used when creating the database sessions (at least in the Model and UserManagement classes)

Are you working on it (since the db_local section exists) or would you prefer a pull request?

Malcom crashes when starting a sniffing session

I have an issue when trying to create a sniffing session.

When I create a sniffer, whether a live one or through a pcap file, the following log appears and no result is shown through the web interface.

Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/mad/Documents/dev/malcom_perso/Malcom/sniffer/netsniffer.py", line 250, in run
    self.load_pcap()
  File "/home/mad/Documents/dev/malcom_perso/Malcom/sniffer/netsniffer.py", line 227, in load_pcap
    self.sniff(stopper=self.stop_sniffing, filter=self.filter, prn=self.handlePacket, stopperTimeout=1, offline=self.engine.setup['SNIFFER_DIR']+"/"+filename)
  File "/home/mad/Documents/dev/malcom_perso/Malcom/sniffer/netsniffer.py", line 653, in sniff
    sel = select([s], [], [], remainStopper)
TypeError: 'module' object is not callable

I'm probably missing something but can't see what. Does an additional module is needed for this to work?

UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1025-1026: invalid continuation byte

DEBUG] [2014-12-18 15:56:05.834409] - (ip analytics for 222.88.195.21)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 238, in run
    self.load_pcap()
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 215, in load_pcap
    self.sniff(stopper=self.stop_sniffing, filter=self.filter, prn=self.handlePacket, stopperTimeout=1, offline=self.engine.setup['SNIFFER_DIR']+"/"+filename)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 658, in sniff
    r = prn(p)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 509, in handlePacket
    self.send_flow_statistics(flow)
  File "/opt/malcom/Malcom/networking/netsniffer.py", line 567, in send_flow_statistics
    self.engine.messenger.broadcast(bson_dumps(data), 'sniffer-data', 'flow_statistics_update')
  File "/opt/malcom/env-malcom/local/lib/python2.7/site-packages/bson/json_util.py", line 125, in dumps
    return json.dumps(_json_convert(obj), *args, **kwargs)
  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1025-1026: invalid continuation byte

API: Edit tags and evil attributes through the API

Elements

Be able to tag an element (add / remove)
Be able to edit an element's evil attribute (by selecting an already existing attribute or adding a new one)

Sniffing sessions

Be able to tag sniffing sessions (add / remove)

Support for newer Ubunut OS's

Would love to see this running natively on ubuntu 18. Thanks Tom!

FT, Time range selection on pcap upload.

It will be nice to be able to select start/stop date of the analysed data on a given pcap.

i cant access my home page in malcom

hi guys,

i installed malcom using docker. but, i cant access my home page of malcom, it directly going into dataset and in feeds all my running services are in "NO". Please guide me in this.

Thanks in advance.

Stealth mode

Add a switch to make sure Malcom does not communicate with external infrastructure:

Prevent rDNS when sniffing and detecting IPs
Prevent DNS resolutions in analytics module
Prevent IP to AS resolutions
Make all the above configurable?

Initiate sniffing session from API

Use the API to initiate a sniffing session. If the request contains a pcap file, use the pcap instead of sniffing from the network.

Error mongodb

Hello,

After run this command ./malcom.py -c malcom.conf I have this error

Detected interfaces:
eth3: 192.168.74.146
eth2: Not defined
eth1: 192.168.74.144
eth0: 192.168.74.148
[+] Starting sniffer...
[+] Successfully loaded sniffer directory: /root/malcom/Malcom/sniffer
[+] Starting TLS proxy on port 9000
Traceback (most recent call last):
File "./malcom.py", line 79, in
setup.sniffer_engine = netsniffer.SnifferEngine(setup, yara_rules=yara_rules)
File "/root/malcom/Malcom/networking/netsniffer.py", line 56, in init
self.model = Model()
File "/root/malcom/Malcom/model/model.py", line 39, in init
self._connection = MongoClient()
File "/root/malcom/env-malcom/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 377, in init
raise ConnectionFailure(str(e))
pymongo.errors.ConnectionFailure: [Errno 111] Connection refused

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.