anssi-fr / polichombr Goto Github PK
View Code? Open in Web Editor NEWCollaborative malware analysis framework
License: Other
Collaborative malware analysis framework
License: Other
When submitting a dummy Yara rule,
the user is not aware of the failure of the rule creation.
ERROR in yara_rule [./poli/controllers/yara_rule.py:149]:
yara.compile(source=raw_data)
SyntaxError: undefined identifier "toto"
There should be an arborescent structure to improve the family view.
For exemple, there should be something like
-> Family 1
---> Family 1.1
-> Family 2
instead of all the families and subfamilies on the same level.
Hi,
When the connectivity fails when viewing the "func Infos" tab, (i.e Error during request, retrying / Closing connection / Connecting using simple HTTP), skelenox still fails : poli_request, res = self.h_conn.getresponse() triggers an httplib.BadStatusLine: '' exception.
Not sure why. Actually the sample is pretty big (>50k subs), and I've got an nginx setup, maybe related ?
Cheers,
The synchronization mechanism in skelenox pushes the already defined names one by one,
which can take a long time for big samples.
Submitting a batch of names could be faster.
There is two problems
Attached two stack traces.
analysis_tools/AnalyzeIt.rb:949:in `block in <main>': undefined local variable or method `egexStr' for main:Object (NameError)
polichombr/metasm/metasm/os/main.rb:165:in `index': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
polichombr/metasm/metasm/os/main.rb:165:in `index
from analysis_tools/AnalyzeIt.rb:757:in `block (2 levels) in <main>'
from analysis_tools/AnalyzeIt.rb:753:in `each'
from analysis_tools/AnalyzeIt.rb:753:in `block in <main>'
from analysis_tools/AnalyzeIt.rb:750:in `each'
from analysis_tools/AnalyzeIt.rb:750:in `<main>'
It is currently cumbersome to store the data from an existing IDB,
a script should be made for this.
Good evening,
I just cloned the repository, and tried to pass the tests as described in "CONTRIBUTING.md". All tests fail, I have a "Working outside of application context." for each of them. The errors correspond to the calls to poli.db in the setUp and tearDown methods of tests_api.py and tests_webui.py. In order to successfully pass the tests I have to enclose the poli.db calls with a "with poli.app.app_context():".
See my comit: https://github.com/harfangeek/polichombr/commit/4367c88ca74d357497732f8a92fbbd74d9974500
Do you experiment this issue?
Thank you.
The proposed names view doesn't permit the user to rename functions with the new names.
Hi folks,
I am trying to install this and getting an error while running the db_create.py script. Following is an error.
line 2, in from poli import db (no module named poli)
any help?
Thanks
Modules are currently autodiscovered in the tasks folder, but enabling or disabling them using a config file could be useful.
Hi,
Disclaimer : we did not actually test it so that's an hypothetic issue (didn't want to pollute the production platform).
As addresses are Virtual addresses and not relatives ones, an IDA program rebase may probably trigger an undefined behavior (dupplicated names, addresses not found, etc.). Actions performed while debugging* (ASLR) might also cause issues.
Anyway, using relative virtual addresses might be a good solution ("just" sub the OEP address ?).
Cheers,
A TypeError
is raised by the SQLAlchemy backend, wich is complaining about SQLite accepting only
Python datetimeobject.
When a renaming conflict happens in skelenox (ie another person has renamed something wich is already renamed in the local IDB), a popup should be shown so the user can choose wether to keep the old name,
or rename with the new one.
Relationships between samples, as for example "dropper -> dropped" or "dropper-> decoy" should be implemented.
Some API endpoints can be very verbose,
they should be paginated in order to extract only meaningful data.
The main endpoints are the following:
The API documentation is not up to date, and the rest of the doc could use some freshening and additions.
Hi,
The "func infos" tab may be quite large when few samples are present, it could be useful to have a sorting capability.
Also, it may be also useful to hide already renamed subs (i.e not starting with "sub_") in order to gain more space.
Cheers,
This is related to #132 , but the SkelIDPHook
, inherited from idaapi.IDP_Hooks
doesn't work
with the new IDAPython API according to the changelog.
Some users don't want the user interface, and there may be compatibility problems with older
IDA versions (eg PyQt5 vs PySide )
Hi,
The commands pushed by Skelenox are displayed as authored by "Analyst" in the Web UI, even if the skelsettings "username" is set to another value.
Cheers,
Skelenox is not compatible with IDA 7.0 without the idaapi compatibility layer (documented here).
A porting should be done to avoid problems with the next IDA update.
Hi !
Adding a "re-perform analysis" button on the Web Interface (in case of server crashs / re-automatic classification / etc.) might be useful :)
Cheers,
The family tab takes a long time to load.
This could be caused by a performance issue in the get_users_for_family function, wich should be replaced by something more efficient.
Very nice and interesting tool. We will have a look to integrate it with MISP like we did with viper.
On the side note, MISP galaxies contain machine parsable information about threat-actors and attacker tools. This could be a nifty extension for the users of your tools to get automatically potential information for classifying their analysis with existing taxonomies.
https://github.com/MISP/misp-galaxy/blob/master/elements/adversary-groups.json
or
https://github.com/MISP/misp-galaxy/blob/master/elements/threat-actor-tools.json
When trying to use skelenox with Ida Pro 6.95 on the mac, there are a few issues:
1: Port number
Port number as defined in skelsettings.json yields a unicode string. on the mac this triggers a bug:
[11/03/2017 10:26] [ERROR] [MainThread]: The polichombr server seems down
Traceback (most recent call last):
File "/scripts/skelenox/skelenox.py", line 148, in get_online
self.__do_init()
File "/scripts/skelenox/skelenox.py", line 164, in __do_init
self.h_conn.connect()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 832, in connect
self.timeout, self.source_address)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 557, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
error: getaddrinfo() argument 2 must be integer or string
Solution is to use "poli_port": 5000, instead of "poli_port": "5000",
2: communication with polichombr
when loading the script in ida, the following occurs:
[12/03/2017 05:55] [ERROR] [MainThread]: HTTPS is not managed at the moment...
[12/03/2017 05:55] [INFO] [MainThread]: Skelenox init finished
[12/03/2017 05:55] [ERROR] [SkelSyncAgent]: The GET request didn't go as expected
[12/03/2017 05:55] [ERROR] [SkelSyncAgent]:
Traceback (most recent call last):
File "/scripts/skelenox/skelenox.py", line 892, in run
self.sync_names()
File "/scripts/skelenox/skelenox.py", line 833, in sync_names
comments = self.skel_conn.get_comments(timestamp=self.last_timestamp)
File "/scripts/skelenox/skelenox.py", line 329, in get_comments
res = self.poli_get(endpoint)
File "/scripts/skelenox/skelenox.py", line 216, in poli_get
result = self.poli_request(endpoint, data, method='GET')
File "/scripts/skelenox/skelenox.py", line 208, in poli_request
raise IOError
IOError
On the server side of things, with debugging turned on, the following is logged:
ERROR in webui [/web/polichombr/poli/views/webui.py:58]:
404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
--------------------------------------------------------------------------------
192.168.2.55 - - [11/Mar/2017 17:37:38] "GET /2/comments/?timestamp=1970-01-01T01:00:00.000000 HTTP/1.1" 404 -
I haven't been able to track this down yet.
Actually when getting idaactions from the API, the timestamp argument is ineffective.
Example: https://github.com/ANSSI-FR/polichombr/blob/master/poli/views/apiview.py#L340
As skelenox pulls a lot of information,
several steps of improvements are doable:
E.g. this file analysis_tools/disassfunc.rb could be done using r2pipe.rb and radare2.
So users wouldn't need to install IDA Pro to get this working.
The current model based on matches such as iat_hash
and machoc_hash
are not sufficient
to explain the links possible between samples.
An example could be to declare a relationship between a dropper and its payload, or a relationship between an orchestrator and its plugins.
It seems that the repository is missing critical tools such as analysis scripts.
Hello,
Is it possible to import several YARA rules at the same time ?
For what I see, it seems we have to import them one by one ...
Thanks !
A screenshot demonstrating binary matching and diffing could be useful
The UI needs several improvements.
Here is a list of analysis modules to implement
Hi !
Sending "sub-information" (address, and machoc hash) from Skelenox may be useful in several cases :
A murmurhash3 python implementation may be found here : https://github.com/wc-duck/pymmh3
Cheers,
There is currently no API endpoint for creating yara rules
Hi,
A Skelenox HTTPS support may be useful. httplib exposes the HTTPSConnection method, SSL certificate pinning may be performed by loading it directly from the configuration file for instance.
Cheers,
In the post data, the API should be able to use a custom TLP level instead of the default TLP-Amber one.
Hi,
While patching skelenox.py, I did load it multiple times in my IDA Pro instance. All of the instances were actually "working" (i.e hooks receivinig notifications), which lead to several issues (multiple requests, connection error, etc.).
Skelenox should find existing instances and safely unload them (i.e remove hooks) in order to avoid such issues. In the main proc, just check for the "skel" object, and if it exists, call its "end_skelenox()" method, then overwrite it.
Cheers,
Bonjour lorsque je lance run.py j'ai ces erreurs :
neolex@archlinux> polichombr$./run.py
INFO in analysis [/home/atlas/polichombr/app/controllers/analysis.py:29]:
Loading tasks
ERROR in analysis [/home/atlas/polichombr/app/controllers/analysis.py:51]:
Could not load task_yara : Parent module 'app.controllers.tasks' not loaded, cannot perform relative import
ERROR in analysis [/home/atlas/polichombr/app/controllers/analysis.py:51]:
Could not load task_peinfo : Parent module 'app.controllers.tasks' not loaded, cannot perform relative import
ERROR in analysis [/home/atlas/polichombr/app/controllers/analysis.py:51]:
Could not load task_analyzeitrb : Parent module 'app.controllers.tasks' not loaded, cannot perform relative import
ERROR in analysis [/home/atlas/polichombr/app/controllers/analysis.py:51]:
Could not load task_strings : Parent module 'app.controllers.tasks' not loaded, cannot perform relative import
- Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
* Restarting with stat
INFO in analysis [/home/atlas/polichombr/app/controllers/analysis.py:29]:
Loading tasks
ERROR in analysis [/home/atlas/polichombr/app/controllers/analysis.py:51]:
Could not load task_yara : Parent module 'app.controllers.tasks' not loaded, cannot perform relative import
ERROR in analysis [/home/atlas/polichombr/app/controllers/analysis.py:51]:
Could not load task_peinfo : Parent module 'app.controllers.tasks' not loaded, cannot perform relative import
ERROR in analysis [/home/atlas/polichombr/app/controllers/analysis.py:51]:
Could not load task_analyzeitrb : Parent module 'app.controllers.tasks' not loaded, cannot perform relative import
ERROR in analysis [/home/atlas/polichombr/app/controllers/analysis.py:51]:
Could not load task_strings : Parent module 'app.controllers.tasks' not loaded, cannot perform relative import
- Debugger is active!
- Debugger pin code: 132-071-423
`
Et lorsque je m'inscrit sur l'interface web :
`builtins.TypeError
TypeError: Unicode-objects must be encoded before hashing
Traceback (most recent call last)
File "/usr/lib/python3.5/site-packages/flask/app.py", line 2000, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/lib/python3.5/site-packages/flask/app.py", line 1991, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/lib/python3.5/site-packages/flask/app.py", line 1567, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/lib/python3.5/site-packages/flask/app.py", line 1988, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.5/site-packages/flask/app.py", line 1641, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.5/site-packages/flask/app.py", line 1544, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/lib/python3.5/site-packages/flask/app.py", line 1639, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.5/site-packages/flask/app.py", line 1625, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/atlas/polichombr/app/views/webui.py", line 127, in register_user
registration_form.poke_id.data)
File "/home/atlas/polichombr/app/controllers/user.py", line 27, in create
apikey_seed = apikey_seed + sha256(username).hexdigest()
TypeError: Unicode-objects must be encoded before hashing
`
Hello,
First, thank you for your work.
I would like to warn you, that by following the INSTALL_DOCKER.md guidelines, the docker container starts sucessfully, but the database does not seem to be created. It is probably because of the commented lines at the end of the Dockerfile (probably for a good reason!)
Here is the the backtrace:
File "/opt/polichombr/flask/lib/python2.7/site-packages/sqlalchemy/pool.py", line 449, in init
self.connection = self.__connect()
File "/opt/polichombr/flask/lib/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect
connection = self.__pool._invoke_creator(self)
File "/opt/polichombr/flask/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
return dialect.connect(_cargs, *_cparams)
File "/opt/polichombr/flask/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect
return self.dbapi.connect(_cargs, *_cparams)
OperationalError: (sqlite3.OperationalError) unable to open database file
Another problem is the fact that ./utils/db_create.py requires the module poli that is only locally referenced (out of the box) at the root folder.
Therefore, we get the error :
Traceback (most recent call last):
File "./utils/db_create.py", line 2, in
from poli import db
ImportError: No module named poli
Regards,
With the current implementation of the set_pass
method in the user
controller ( here ),
the password is never changed even if the user changes it in the user view.
Instead of checking if the file exists in db_create.py,
we should handle gracefully the DatabaseAlreadyControlledError
This is from a fresh GIT installed.
There is an exception when uploading a zip file
File "poli/controllers/api.py", line 74, in create_sample_and_run_analysis
with zipfile.ZipFile(file_data, "r") as zcl:
File "/usr/lib/python2.7/zipfile.py", line 756, in __init__
self.fp = open(file, modeDict[mode])
TypeError: file() argument 1 must be encoded string without null bytes, not str
Hi,
This is a minor bug : when I try to "machoc-compare" a sample I'm redirected to the sample's home page, not the "#matches" one. Results are correct, this is only a "UI" issue :)
Cheers,
A family name is not editable after having been created.
A new form should be added to handle this case
This bug can be triggered when running the tests ( tests/full_tests.py
),
when the main db (app.db file) is not created.
The tests will then give you a lots of SQLAlchemyOperationalError
about the inexistant Sample
table
Hi,
It's actually impossible to update family-related data (files / detection items). It can actually be performed by deleting and re-creating the item, so that's only a minor update request regarding the time required to implement the features, but may be useful to do it one day :)
Cheers,
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.