The fame from certsocietegenerale

how to connect modules with fame

Description

[how to connect modules like apk verification, cuckoo, cuckoo modified, joe, and office_macros with fame, thanks.]

Steps to Reproduce

[.]

Expected behavior

[Fame can use available modules like apk verification, cuckoo, cuckoo modified, joe, and office_macros and other]

Actual behavior

[fame can not use the available modules like apk verification, cuckoo, cuckoo modified, joe, and office_macros and other.]

Debug

[root@bismillah-VirtualBox:~/fame# utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-4.10.0-28-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12

########## DEPENDENCIES ###########

alabaster==0.7.10
amqp==2.3.1
Babel==2.6.0
billiard==3.5.0.3
celery==4.1.1
certifi==2018.4.16
chardet==3.0.4
click==6.7
docutils==0.14
Flask==1.0.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
gitdb2==2.0.3
GitPython==2.1.10
idna==2.6
imagesize==1.0.0
itsdangerous==0.24
Jinja2==2.10
kombu==4.2.0
LEPL==5.1.3
markdown2==2.3.5
MarkupSafe==1.0
packaging==17.1
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-magic==0.4.15
pytz==2018.4
requests==2.18.4
rfc6266==0.0.4
six==1.11.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.7.5
sphinx-rtd-theme==0.3.1
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.0.1
typing==3.6.4
urllib3==1.22
vine==1.1.4
Werkzeug==0.14.1
zxcvbn==1.0
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

########## MongoDB ##########

Version: 3.6.4
Authorization check: True

########## Configuration ##########

types: True
virustotal: True
email: False
malware_config: False
volatility: True

Modules:

McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Enabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Enabled Configured
cuckoo Processing Enabled Configured
cuckoo_modified Processing Enabled Configured
cutthecrap Processing Disabled Not Configured
eml Processing Enabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Disabled Not Configured
office_macros Processing Enabled Configured
pdf Processing Enabled Configured
rat_decoders Processing Enabled Configured
url_download Processing Enabled Configured
zip Processing Enabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
kvm Virtualization Disabled Configured
virtualbox Virtualization Disabled Configured
root@bismillah-VirtualBox:~/fame#

]

Russian symbols mess

Description

If module returns Russian symbols in detailed results, we see a mess

Steps to Reproduce

Write sample module, that will return string "фывапролдж"

Expected behavior

see "фывапролдж" in web interface

Actual behavior

Like this:

Debug

fame@fame-server:~/fame$ utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-4.4.0-83-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12

########## DEPENDENCIES ###########

alabaster==0.7.10
amqp==2.2.1
androguard==3.0.1
Babel==2.4.0
bamfdetect==1.6.13
beautifulsoup4==4.6.0
billiard==3.5.0.3
bs4==0.0.1
celery==4.0.2
certifi==2017.4.17
chardet==3.0.4
click==6.7
docutils==0.13.1
Flask==0.12.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.0
future==0.16.0
gitdb2==2.0.2
GitPython==2.1.5
googleplay-api==0.1.0
idna==2.5
ijson==2.3
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.6
kombu==4.0.2
LEPL==5.1.3
markdown2==2.3.4
MarkupSafe==1.0
oletools==0.51
pbkdf2==1.3
pefile==2016.3.28
pkg-resources==0.0.0
protobuf==3.3.0
pycrypto==2.6.1
pyelftools==0.24
Pygments==2.2.0
pymongo==3.4.0
python-magic==0.4.13
pytz==2017.2
rarfile==3.0
requests==2.18.1
rfc6266==0.0.4
six==1.10.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.6.3
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
sphinxcontrib-websupport==1.0.1
typing==3.6.1
urllib3==1.21.1
uWSGI==2.0.15
vine==1.1.4
volatility==2.6
Werkzeug==0.12.2
yara-python==3.6.3
zxcvbn==1.0

########## MongoDB ##########

Version: 3.4.6
Authorization check: True

########## Configuration ##########

types: True
virustotal: False
email: False
malware_config: False
volatility: True

Modules:

McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Disabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Enabled Configured
cuckoo Processing Enabled Configured
cuckoo_modified Processing Disabled Configured
cutthecrap Processing Disabled Not Configured
eml Processing Enabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Disabled Not Configured
office_macros Processing Enabled Configured
pdf Processing Enabled Configured
rat_decoders Processing Enabled Configured
url_download Processing Enabled Configured
zip Processing Enabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
kvm Virtualization Disabled Configured
virtualbox Virtualization Disabled Configured

No initial user account asked.

Description

Where to enter the initial user account

Steps to Reproduce

Install from git for a development environment

Expected behavior

The administrator guide describes a user account entered during setup process.

Actual behavior

No user email or account provided.

Debug

########## VERSION ##########

OS: Linux-4.9.16-1-lts-x86_64-with-glibc2.2.5
Python: 2.7.13

########## DEPENDENCIES ###########

alabaster==0.7.10
amqp==2.1.4
androguard==3.0
appdirs==1.4.3
Babel==2.3.4
bamfdetect==1.6.13
beautifulsoup4==4.5.3
billiard==3.5.0.2
bs4==0.0.1
celery==4.0.2
click==6.7
docutils==0.13.1
Flask==0.12
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.4.5
future==0.16.0
gitdb2==2.0.0
GitPython==2.1.3
googleplay-api==0.1.0
ijson==2.3
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.5
kombu==4.0.2
markdown2==2.3.3
MarkupSafe==1.0
oletools==0.50
packaging==16.8
pbkdf2==1.3
pefile==2016.3.28
protobuf==3.2.0
pycrypto==2.6.1
Pygments==2.2.0
pymongo==3.4.0
pyparsing==2.2.0
python-magic==0.4.13
pytz==2016.10
rarfile==3.0
requests==2.13.0
six==1.10.0
smmap2==2.0.1
snowballstemmer==1.2.1
Sphinx==1.5.3
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
vine==1.1.3
volatility==2.6
Werkzeug==0.12.1
yara-python==3.5.0
zxcvbn==1.0

########## MongoDB ##########

Version: 3.4.2
Authorization check: True

########## Configuration ##########

Traceback (most recent call last):
File "utils/troubleshoot.py", line 83, in
main()
File "utils/troubleshoot.py", line 80, in main
configuration()
File "utils/troubleshoot.py", line 58, in configuration
for config in Config.find():
File "/home/tnormand/Repositories/github/fame/fame/common/mongo_dict.py", line 24, in find
for obj in objs:
File "/home/tnormand/Repositories/github/fame/env/lib/python2.7/site-packages/pymongo/cursor.py", line 1114, in next
if len(self.__data) or self._refresh():
File "/home/tnormand/Repositories/github/fame/env/lib/python2.7/site-packages/pymongo/cursor.py", line 1036, in _refresh
self.__collation))
File "/home/tnormand/Repositories/github/fame/env/lib/python2.7/site-packages/pymongo/cursor.py", line 928, in __send_message
helpers._check_command_response(doc['data'][0])
File "/home/tnormand/Repositories/github/fame/env/lib/python2.7/site-packages/pymongo/helpers.py", line 210, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: not authorized on fame to execute command { find: "settings", filter: {} }

SAML?

Hey All,

Anyone get SAML working successfully with Fame?

Thx,

Jim

problem fame

Description

[when I managed to access the fame, I got into trouble with the detail analysis menu, where when I tried to do the analysis I only get some menu just like: file details, execution path, logs and dimenu path there is still some constraint, when I read in fame documentation there some menu options such as: execution path, observables, extracttions, detailed results, and logs.
roughly for that problem how is the solution?
Your help is very much needed, thank you.]

Steps to Reproduce

[Describe the steps to reproduce]

Expected behavior

[How are you expecting the application to behave?]

Actual behavior

[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]

Debug

[Include the output of utils/run.sh utils/troubleshoot.py]

Ansible playbook for Setup automation

Description

I guess this project is being used by a lot of people and it would be nice to have one line installer script or ansible playbook to automate entire installation and configuration process.
Its kind of a feature request, but if it is in scope then I can write it and send a PR

trouble update modules repository

Description

[excusme, i have trouble from update module repositori, and i have message : ValueError: Reserved characters such as ':' must be escaped according RFC 2396. An IPv6 address literal must be enclosed in '[' and ']' according to RFC 2732.

can you help me, thankyou.]

Steps to Reproduce

[Describe the steps to reproduce]

Expected behavior

[How are you expecting the application to behave?]

Actual behavior

[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]

Debug

[Include the output of utils/run.sh utils/troubleshoot.py]

Production Instructions Not Working

Description

I am repeatedly receiving errors for the community modules when following the production installation instructions.

Steps to Reproduce

apt-get -qq update
apt-get install git python-pip python-dev p7zip-full
pip install virtualenv
pip install --upgrade pip
cp -v /usr/local/bin/pip /usr/bin/pip
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
apt-get -qq update
apt-get install -y mongodb-org
systemctl enable mongod
systemctl start mongod
mongo

> use admin
> db.createUser({ user: "admin", pwd: "example", roles: [ { role: "userAdminAnyDatabase", db: "admin" } ] })
> use fame
> db.createUser({ user: "fame", pwd: "example", roles: [ { role: "dbOwner", db: "fame" } ] })

/etc/mongod.conf

security:
  authorization: enabled

systemctl restart mongod
git clone https://github.com/certsocietegenerale/fame
cd fame
pip install uwsgi
/etc/systemd/system/fame_web.service

[Unit]
Description=FAME web server

[Service]
Type=simple
ExecStart=/bin/bash -c "cd /fame && uwsgi -H /fame/env --uid root --gid root --socket /tmp/fame.sock --chmod-socket=660 --chown-socket root:www-data -w webserver --callable app"

[Install]
WantedBy=multi-user.target

/etc/systemd/system

[Unit]
Description=FAME workers

[Service]
Type=simple
User=root
ExecStart=/bin/bash -c 'cd /fame && utils/run.sh worker.py'

[Install]
WantedBy=multi-user.target

utils/run.sh utils/install.py
systemctl enable fame_web
systemctl enable fame_worker
systemctl start fame_web
systemctl start fame_worker
apt-get install nginx
rm /etc/nginx/sites-enabled/default
/etc/nginx/sites-available/fame

upstream fame {
    server unix:///tmp/fame.sock;
}

server {
    listen 80 default_server;

    # Allows big file upload
    client_max_body_size 0;

    location / {
      include uwsgi_params;
      uwsgi_pass fame;
    }

    location /static/ {
      alias /fame/web/static/;
    }
}

ln -s /etc/nginx/sites-available/fame /etc/nginx/sites-enabled/fame
systemctl restart nginx

Expected behavior

All modules should not return any errors and should be in a usable state.

Actual behavior

apk_verification, cuckoo, cuckoo_modified, joe, and office_macros all return EnvironmentErrors.

/fame/fame/modules/community/processing/apk_verification/requirements.txt: error on 'circle':

Requirement already satisfied: androguard in ./env/lib/python2.7/site-packages (from -r /fame/fame/modules/community/processing/apk_verification/requirements.txt (line 1)) (3.2.1)
Collecting googleplay-api (from -r /fame/fame/modules/community/processing/apk_verification/requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/d2/74/e089b2d8b9caf88c7738f631a22fee675db6f17d05df9a9ceec99117f601/googleplay_api-0.1.0.tar.gz
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-jACyBx/4bd56651a47a3f4a642aaa5b8ab809c8219f0c6e035325978b12dcc6'

/fame/fame/modules/community/processing/cuckoo/requirements.txt: error on 'circle':

Collecting ijson (from -r /fame/fame/modules/community/processing/cuckoo/requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/7f/e9/8508c5f4987ba238a2b169e582c1f70a47272b22a2f1fb06b9318201bb9e/ijson-2.3-py2.py3-none-any.whl
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-jACyBx/af051b681629ce9f0f028b1d3e10f6238379d2a9b8f081e8705190c0'

/fame/fame/modules/community/processing/cuckoo_modified/requirements.txt: error on 'circle':

Requirement already satisfied: requests in ./env/lib/python2.7/site-packages (from -r /fame/fame/modules/community/processing/cuckoo_modified/requirements.txt (line 1)) (2.18.4)
Collecting ijson (from -r /fame/fame/modules/community/processing/cuckoo_modified/requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/7f/e9/8508c5f4987ba238a2b169e582c1f70a47272b22a2f1fb06b9318201bb9e/ijson-2.3-py2.py3-none-any.whl
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-jACyBx/af051b681629ce9f0f028b1d3e10f6238379d2a9b8f081e8705190c0'

/fame/fame/modules/community/processing/joe/requirements.txt: error on 'circle':

Requirement already satisfied: requests in ./env/lib/python2.7/site-packages (from -r /fame/fame/modules/community/processing/joe/requirements.txt (line 1)) (2.18.4)
Collecting ijson (from -r /fame/fame/modules/community/processing/joe/requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/7f/e9/8508c5f4987ba238a2b169e582c1f70a47272b22a2f1fb06b9318201bb9e/ijson-2.3-py2.py3-none-any.whl
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-jACyBx/af051b681629ce9f0f028b1d3e10f6238379d2a9b8f081e8705190c0'

/fame/fame/modules/community/processing/office_macros/requirements.txt: error on 'circle':

Collecting oletools (from -r /fame/fame/modules/community/processing/office_macros/requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/79/f5/9b1a89145ac9bce77c235fee549fc7af617d778bb29af4c8dd1561813a10/oletools-0.53.1.zip
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-jACyBx/1242e770b9a28f026a448edb32b189c35bb53e95ba2bae9227a930a0'

bamfdetect returns an AttributeError.

python /fame/fame/modules/community/processing/bamfdetect/install.py: error on 'circle':

Traceback (most recent call last):
  File "/fame/fame/modules/community/processing/bamfdetect/install.py", line 8, in <module>
    main()
  File "/fame/fame/modules/community/processing/bamfdetect/install.py", line 5, in main
    pip.main(['install', '--no-deps', 'git+https://github.com/bwall/bamfdetect#egg=BAMF_Detect'])
AttributeError: 'module' object has no attribute 'main'

The other modules do not immediately return an error and I have not tested all of them. The pdf module returns a KeyError, though, when ran against a tested PDF file.

2018-07-24 15:44: debug: Trying to queue module 'pdf'
2018-07-24 15:44: debug: Trying to run pdf
2018-07-24 15:44: error: pdf: Could not run on /fame/storage/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513/eicar.pdf.
 Traceback (most recent call last):
  File "/fame/fame/core/module.py", line 492, in _try_each
    return self.each_with_type(target, file_type)
  File "/fame/fame/core/module.py", line 450, in each_with_type
    return self.each(target)
  File "/fame/fame/modules/community/processing/pdf/pdf.py", line 41, in each
    if element['vuln_cve_list']:
KeyError: 'vuln_cve_list'

2018-07-24 15:44: debug: Done with pdf

Debug

root@circle:/fame# utils/run.sh utils/troubleshoot.py 
[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-4.13.0-36-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12


########## DEPENDENCIES ###########

alabaster==0.7.11
amqp==2.3.2
androguard==3.2.1
asn1crypto==0.24.0
Babel==2.6.0
backports.functools-lru-cache==1.5
backports.shutil-get-terminal-size==1.0.0
billiard==3.5.0.4
celery==4.1.1
certifi==2018.4.16
chardet==3.0.4
click==6.7
colorama==0.3.9
cycler==0.10.0
decorator==4.3.0
docutils==0.14
enum34==1.1.6
Flask==1.0.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
future==0.16.0
gitdb2==2.0.4
GitPython==2.1.11
idna==2.6
imagesize==1.0.0
ipython==5.7.0
ipython-genutils==0.2.0
itsdangerous==0.24
Jinja2==2.10
kiwisolver==1.0.1
kombu==4.2.1
LEPL==5.1.3
lxml==4.2.3
markdown2==2.3.5
MarkupSafe==1.0
matplotlib==2.2.2
networkx==2.1
numpy==1.15.0
packaging==17.1
pathlib2==2.3.2
pexpect==4.6.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
ptyprocess==0.6.0
pyelftools==0.24
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-dateutil==2.7.3
python-magic==0.4.15
pytz==2018.5
requests==2.18.4
rfc6266==0.0.4
scandir==1.7
simplegeneric==0.8.1
six==1.11.0
smmap2==2.0.4
snowballstemmer==1.2.1
Sphinx==1.7.6
sphinx-rtd-theme==0.3.1
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.1.0
subprocess32==3.5.2
traitlets==4.3.2
typing==3.6.4
urllib3==1.22
vine==1.1.4
volatility==2.6
wcwidth==0.1.7
Werkzeug==0.14.1
yara-python==3.7.0
zxcvbn==1.0


########## MongoDB ##########

Version: 4.0.0
Authorization check: True

########## Configuration ##########

types: True
virustotal: False
email: False
malware_config: False
volatility: True

Modules:

McAfee                    Antivirus            Disabled   Configured     
Sophos                    Antivirus            Disabled   Configured     
Symantec                  Antivirus            Disabled   Not Configured 
apk                       Processing           Disabled   Configured     
apk_verification          Processing           Disabled   Not Configured 
bamfdetect                Processing           Disabled   Configured     
cuckoo                    Processing           Disabled   Configured     
cuckoo_modified           Processing           Disabled   Configured     
cutthecrap                Processing           Disabled   Not Configured 
eml                       Processing           Disabled   Configured     
joe                       Processing           Disabled   Not Configured 
marcher_config            Processing           Disabled   Configured     
mem_yara                  Processing           Disabled   Not Configured 
office_macros             Processing           Disabled   Configured     
pdf                       Processing           Disabled   Configured     
rat_decoders              Processing           Disabled   Configured     
url_download              Processing           Disabled   Configured     
zip                       Processing           Disabled   Configured     
slack                     Reporting            Disabled   Not Configured 
Yeti                      Threat Intelligence  Disabled   Not Configured 
kvm                       Virtualization       Disabled   Configured     
virtualbox                Virtualization       Disabled   Configured

Adding possibility to cancel/remove/manage analysis

Hello,

Do you plan to add some capacities to delete/cancel/rerun analysis ?
In some case (when misconfiguration) some tasks are staying in 'pending' status. There is a way to delete them or rerun them ?

Best regards,

FIND WEB OUTPUT

web
web/views
web/views/mixins.py
web/views/helpers.pyc
web/views/constants.pyc
web/views/analyses.py
web/views/analyses.pyc
web/views/users.pyc
web/views/modules.pyc
web/views/search.pyc
web/views/negotiation.py
web/views/constants.py
web/views/init.pyc
web/views/negotiation.pyc
web/views/files.pyc
web/views/search.py
web/views/init.py
web/views/configs.py
web/views/configs.pyc
web/views/modules.py
web/views/helpers.py
web/views/mixins.pyc
web/views/files.py
web/views/users.py
web/init.pyc
web/init.py
web/static
web/static/js
web/static/js/bootstrap-notify.min.js
web/static/js/bootstrap.min.js
web/static/js/fame.js
web/static/js/tagsinput.js
web/static/js/checkbox.js
web/static/js/bootstrap.js
web/static/js/typeahead.bundle.min.js
web/static/js/template.js
web/static/js/npm.js
web/static/js/highlight.min.js
web/static/js/fileinput.min.js
web/static/js/modules-typeahead.js
web/static/js/jquery-1.11.3.min.js
web/static/js/fame-file.js
web/static/js/handlebars.min.js
web/static/fonts
web/static/fonts/Pe-icon-7-stroke.woff
web/static/fonts/Pe-icon-7-stroke.eot
web/static/fonts/fontawesome-webfont.svg
web/static/fonts/fontawesome-webfont.woff2
web/static/fonts/glyphicons-halflings-regular.ttf
web/static/fonts/Pe-icon-7-stroke.svg
web/static/fonts/glyphicons-halflings-regular.woff2
web/static/fonts/Pe-icon-7-stroke.ttf
web/static/fonts/glyphicons-halflings-regular.eot
web/static/fonts/fontawesome-webfont.ttf
web/static/fonts/fontawesome-webfont.woff
web/static/fonts/fontawesome-webfont.eot
web/static/fonts/glyphicons-halflings-regular.woff
web/static/fonts/glyphicons-halflings-regular.svg
web/static/fonts/FontAwesome.otf
web/static/favicon.png
web/static/img
web/static/img/computer_code.jpg
web/static/img/sidebar.jpg
web/static/img/mask.png
web/static/img/avatars
web/static/img/avatars/58da87c8fb99aa13c2f5c47d.png
web/static/img/avatars/default.png
web/static/css
web/static/css/bootstrap.css
web/static/css/fileinput.min.css
web/static/css/bootstrap-theme.css
web/static/css/bootstrap-theme.min.css
web/static/css/template.css
web/static/css/bootstrap.css.map
web/static/css/pe-icon-7-stroke.css
web/static/css/font-awesome.min.css
web/static/css/animate.css
web/static/css/bootstrap-theme.css.map
web/static/css/bootstrap.min.css
web/static/css/fame.css
web/static/css/highlight.railscasts.min.css
web/auth
web/auth/user_password
web/auth/user_password/user_management.py
web/auth/user_password/views.pyc
web/auth/user_password/views.py
web/auth/user_password/init.pyc
web/auth/user_password/init.py
web/auth/user_password/templates
web/auth/user_password/templates/password_reset_form.html
web/auth/user_password/templates/mail_user_creation.html
web/auth/user_password/templates/mail_reset_password.html
web/auth/user_password/templates/auth_profile.html
web/auth/user_password/templates/login.html
web/auth/user_password/templates/base_unauthenticated.html
web/auth/user_password/templates/password_reset.html
web/auth/user_password/user_management.pyc
web/auth/saml
web/auth/saml/user_management.py
web/auth/saml/views.py
web/auth/saml/init.py
web/auth/saml/config
web/auth/saml/config/init.py
web/auth/saml/config/.gitignore
web/auth/saml/config/custom_mappings.py.sample
web/auth/saml/config/settings.json.sample
web/auth/init.pyc
web/auth/init.py
web/templates
web/templates/configs
web/templates/configs/single_block.html
web/templates/configs/target_attributes.html
web/templates/configs/botnet_list.html
web/templates/configs/index.html
web/templates/configs/show.html
web/templates/base.html
web/templates/users
web/templates/users/_form.html
web/templates/users/new.html
web/templates/users/index.html
web/templates/users/profile.html
web/templates/search.html
web/templates/analyses
web/templates/analyses/details.html
web/templates/analyses/new.html
web/templates/analyses/index.html
web/templates/analyses/show.html
web/templates/analyses/_options.html
web/templates/analyses/list.html
web/templates/files
web/templates/files/details.html
web/templates/files/index.html
web/templates/files/show.html
web/templates/files/list.html
web/templates/modules
web/templates/modules/module_configuration.html
web/templates/modules/templates.html
web/templates/modules/configuration.html
web/templates/modules/index.html
web/templates/modules/repository_new.html
web/templates/modules/_configuration.html

pymongo.errors.OperationFailure:text search not enabled

Description

Installing fame and getting error mentioned after SSH key generates

Steps to Reproduce

makflwana@ubuntu:~/fame$ sudo utils/run.sh utils/install.py
[+] Using existing virtualenv.

[+] Installing requirements ...
[?] MongoDB host [localhost]:
[?] MongoDB port [27017]:
[?] MongoDB database [fame]:

Choose your installation type:

1: Web server + local worker
2: Remote worker

[?] Installation type [1]: 1
[?] FAME's URL for users (e.g. https://fame.yourdomain/): https://fame.localhost.com
[+] Creating configuration file ...
[+] SSH key already exists.
Traceback (most recent call last):
File "utils/install.py", line 227, in
main()
File "utils/install.py", line 221, in main
perform_local_installation(context)
File "utils/install.py", line 144, in perform_local_installation
fame_init()
File "/home/makflwana/fame/fame/core/init.py", line 6, in fame_init
store.connect()
File "/home/makflwana/fame/fame/core/store.py", line 40, in connect
self.files.create_index([("$**", TEXT)], background=True)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/collection.py", line 1529, in create_index
self.__create_index(keys, kwargs)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/collection.py", line 1437, in __create_index
sock_info, index, True, False, False, wcn)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/collection.py", line 562, in _insert
check_keys, manipulate, write_concern, op_id, bypass_doc_val)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/collection.py", line 551, in _insert_one
self.__write_response_codec_options)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/collection.py", line 488, in _legacy_write
rqst_id, msg, max_size, acknowledged)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/pool.py", line 473, in legacy_write
return helpers._check_gle_response(response)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/helpers.py", line 250, in _check_gle_response
raise OperationFailure(details["err"], code, result)
pymongo.errors.OperationFailure: text search not enable

Also, I am testing the FAME installation on VM.

Running analysis failing on module

Description

I get the following error message when trying to execute an analysis by selecting Bamfdetect:

2017-04-01 08:42: error: Could not find execution path to target bamfdetect

Steps to Reproduce

Submit sample file select bamfdetect as analyser and click submit

Expected behavior

Give a result not an error in logs section

Actual behavior

Log section of the analysis shows:

2017-04-01 08:48: error: Could not find execution path to target bamfdetect

Debug

fame@ubuntu:~/fame$ utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-4.4.0-62-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12

########## DEPENDENCIES ###########

alabaster==0.7.10
amqp==2.1.4
appdirs==1.4.3
Babel==2.4.0
billiard==3.5.0.2
celery==4.0.2
click==6.7
docutils==0.13.1
Flask==0.12.1
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.4.5
gitdb2==2.0.0
GitPython==2.1.3
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.5
kombu==4.0.2
markdown2==2.3.3
MarkupSafe==1.0
packaging==16.8
Pygments==2.2.0
pymongo==3.4.0
pyparsing==2.2.0
python-magic==0.4.13
pytz==2017.2
requests==2.13.0
six==1.10.0
smmap2==2.0.1
snowballstemmer==1.2.1
Sphinx==1.5.3
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
vine==1.1.3
Werkzeug==0.12.1
zxcvbn==1.0

########## MongoDB ##########

Version: 3.4.3
Authorization check: True

########## Configuration ##########

types: True
virustotal: False
email: False
malware_config: False
volatility: True

Modules:

McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Disabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Enabled Configured
cuckoo_modified Processing Disabled Configured
eml Processing Disabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Disabled Not Configured
office_macros Processing Disabled Configured
pdf Processing Disabled Configured
url_download Processing Disabled Configured
zip Processing Disabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured

problem in the menu Analysis Details.

Description

[problem in the menu Analysis Details, should there are some menus, for example execution path, observables, extracttions, detail results, and logs and while my menu is only there file details, execution path, logs.]

Steps to Reproduce

[Describe the steps to reproduce]

Expected behavior

[How are you expecting the application to behave?]

Actual behavior

[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]

Debug

[Include the output of utils/run.sh utils/troubleshoot.py]

cannot running "utils/run.sh webserver.py" and "utils/run.sh utils/install.py"

Description

[can you help me, i cannot running "utils/run.sh webserver.py" it says " WARNING: Do not use the development server in a production environment." and "utils/run.sh utils/install.py" it says "Could not connect to MongoDB database", thank you.]

Steps to Reproduce

[Describe the steps to reproduce]

Expected behavior

[How are you expecting the application to behave?]

Actual behavior

[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]

Debug

[Include the output of utils/run.sh utils/troubleshoot.py]

Patch: Include File Size in Object Details and file Class.

Hello,

I didn't see this available in any of the models, so I added it to the file model.

diff --git a/fame/core/file.py b/fame/core/file.py
index 288a478..ab8816b 100755
--- a/fame/core/file.py
+++ b/fame/core/file.py
@@ -124,6 +124,7 @@ class File(MongoDict):
         self['detailed_type'] = magic.from_file(self['filepath'])
         self['mime'] = magic.from_file(self['filepath'], mime=True)
         self['analysis'] = []
+        self['size'] = os.path.getsize(self['filepath'])
 
         # Init antivirus status
         self['antivirus'] = {}

diff --git a/web/templates/files/details.html b/web/templates/files/details.html
index e6d36dc..ce05576 100755
--- a/web/templates/files/details.html
+++ b/web/templates/files/details.html
@@ -26,6 +26,10 @@
                 </div>
                 {% if not file.type == 'url' %}
                 <div class="row">
+                    <div class="col-sm-2 text-right"><strong>File Size</strong></div>
+                    <div class="col-sm-10">{{file.size}} bytes</div>
+                </div>
+                <div class="row">
                     <div class="col-sm-2 text-right"><strong>MD5</strong></div>
                     <div class="col-sm-10">{{file.md5}}</div>
                 </div>

fame installation is not running

Description

[when running the script : "utils/run.sh utils/install.py" i have message 127.0.0.1:27017: [Errno 111] Connection refused
]

Steps to Reproduce

[Describe the steps to reproduce]

Expected behavior

[can do installation fame]

Actual behavior

[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]

Debug

[root@bismillah-VirtualBox:/home/bismillah/fame# utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-4.10.0-28-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12

########## DEPENDENCIES ###########

alabaster==0.7.11
amqp==2.3.2
Babel==2.6.0
billiard==3.5.0.4
celery==4.1.1
certifi==2018.4.16
chardet==3.0.4
click==6.7
docutils==0.14
Flask==1.0.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
gitdb2==2.0.4
GitPython==2.1.11
idna==2.6
imagesize==1.0.0
itsdangerous==0.24
Jinja2==2.10
kombu==4.2.1
LEPL==5.1.3
markdown2==2.3.5
MarkupSafe==1.0
packaging==17.1
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-magic==0.4.15
pytz==2018.5
requests==2.18.4
rfc6266==0.0.4
six==1.11.0
smmap2==2.0.4
snowballstemmer==1.2.1
Sphinx==1.7.6
sphinx-rtd-theme==0.3.1
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.1.0
typing==3.6.4
urllib3==1.22
vine==1.1.4
Werkzeug==0.14.1
zxcvbn==1.0

########## MongoDB ##########

Could not connect to MongoDB: localhost:27017: [Errno 111] Connection refused
]

Install From Scratch - Internal error submitting a sample

Description

When submitting a new sample, the webserver returned an Internal error.

Steps to Reproduce

Install from scratch and submission of the sample Shamoon from theZoo repo

Expected behavior

[How are you expecting the application to behave?]

Actual behavior

20170821-fame-issue_on-submit.txt

Debug

20170821-fame-issue_on-submit_troubleshoot.txt
``]

ThreatIntelligence Module Debugging

I'm writing a TI module to enrich analysis using an internal data source. I implemented both "ioc_submission" and "ioc_lookup" methods. How can I test this now? Nothing in the interface shows up similar to the "Send to Yeti".

It may be convenient to add a way to test TI modules like processing modules "single_module.py".

'Download' should provide a pw protected zip instead of the malware

Description

Feature request

Serve a zip with a simple password would prevent accidentally running malware, when downloading a sample.

remote worker url_download broke

After enabling the url_download module, files were only downloadable via the local fame worker with the server. When a remote worker consumed a task to download from the url, the following error occurred:

2018-10-20 21:26: debug: Trying to queue module 'url_download'
2018-10-20 21:26: debug: Trying to run url_download
2018-10-20 21:26: debug: Adding extracted file '/fame/temp/5bd59cca9a6747959a194db4260430e8/com.parental.control.v4.apk'
2018-10-20 21:26: error: url_download: Could not run on http://xxx.xxx.xxx.xxx:8000/com.parental.control.v4.apk.
 Traceback (most recent call last):
  File "/fame/fame/core/module.py", line 492, in _try_each
    return self.each_with_type(target, file_type)
  File "/fame/fame/core/module.py", line 450, in each_with_type
    return self.each(target)
  File "/fame/fame/modules/community/processing/url_download.py", line 35, in each
    self.add_extracted_file(filepath)
  File "/fame/fame/core/module.py", line 345, in add_extracted_file
    self._analysis.add_extracted_file(location)
  File "/fame/fame/core/analysis.py", line 90, in add_extracted_file
    response = send_file_to_remote(filepath, '/files/')
  File "/fame/fame/common/utils.py", line 74, in send_file_to_remote
    response.raise_for_status()
  File "/fame/env/local/lib/python2.7/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: http://fame:4200/files/

2018-10-20 21:26: debug: Done with url_download

The FAME server shows a traceback that indicates the files.py post() view function crashed because it was trying to render a template. The fame server is trying to render a template that doesn't exist. This is fixed by adding an "Accept: application/json" header into the requests.post call in send_file_to_remote() in utils.py to indicate that the server should respond to the worker with a JSON encoded payload.

Subsequently, if this change is added in, the core/analysis.py file add_extracted_file() function will need to load the response payload as bson so that it can resolve the File object correctly.

common/utils.py

def send_file_to_remote(file, url):
    if isinstance(file, basestring):
        file = open(file, 'rb')

    url = urljoin(fame_config.remote, url)
    response = requests.post(url, files={'file': file}, headers={'X-API-KEY': fame_config.api_key,
                                                                 'Accept': 'application/json'})
    response.raise_for_status()

    file.close()

    return response

core/analysis.py

from bson.json_util import loads
...
    def add_extracted_file(self, filepath):
        self.log('debug', "Adding extracted file '{}'".format(filepath))

        fd = open(filepath, 'rb')
        filename = os.path.basename(filepath)
        f = File(filename=filename, stream=fd, create=False)

        if not f.existing:
            if fame_config.remote:
                response = send_file_to_remote(filepath, '/files/')
                f = File(loads(response.text)['file'])
            else:
                f = File(filename=os.path.basename(filepath), stream=fd)

            f.analyze(self['groups'], self['analyst'], None, self['options'])

        fd.close()

        self.append_to('extracted_files', f['_id'])
        f.add_parent_analysis(self)

Hope this helps! I have tested this and it works in my forked version (feel free to pull in that code if these changes seem reasonable)

Manually canceling analysis jobs

Questions

I submitted a couple of samples on fame for analysis and then manually stopped the jobs at the cuckoo-modified server. Before all this, I modified the cuckoo-modified configuration and disabled the time-out options. Now both analysis on fame are running perpetually.

Is there a way to cancel these jobs on the fame portal? If not, how to cancel them from the shell?

Thanks in advance for the help and for releasing this tool to the public.

remote worker module updates not working

Remote worker's don't appear to be subscribed to the 'updates' queue. When the modules repository is updated from the GUI, it puts the message on the 'messages' Mongo collection, but the remote workers never consume it because of the following code in worker.py:

# A local worker should also take care of updates
    if not fame_config.remote:
        queues.append('updates')

I had to remove the 'if' statement in order to get remote workers to use the 'updates' queue to sync the new repo.

Can fame works on Vmware esxi?

Hi,

Do you know if Fame works on vmware esxi???

Thx

generic question regarding JSON output and misp-object

We did a quick review of fame and we really like the approach, the overall design and the modularity. As we are working on the object model in MISP (to be released soon), we were wondering if the fame format json output as seen in this example.

Will the fame JSON format stable in the forthcoming releases?
Can we base a fame misp-object based on this? or should we expect major changes soon?

Thank you very much

Error while running utils/run.sh utils/troubleshoot.py

Description

Hey,
When I tried to run utils/troubleshoot.py, I got the following output.

Debug

fame@fame:~/fame$ utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-4.4.0-92-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12

########## DEPENDENCIES ###########

alabaster==0.7.10
amqp==2.2.1
androguard==3.0.1
Babel==2.4.0
bamfdetect==1.6.13
beautifulsoup4==4.6.0
billiard==3.5.0.3
bs4==0.0.1
celery==4.1.0
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
docutils==0.14
Flask==0.12.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.0
future==0.16.0
gitdb2==2.0.2
GitPython==2.1.5
googleplay-api==0.1.0
idna==2.6
ijson==2.3
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.6
kombu==4.1.0
LEPL==5.1.3
markdown2==2.3.4
MarkupSafe==1.0
oletools==0.51
pbkdf2==1.3
pefile==2017.8.1
protobuf==3.4.0
pycrypto==2.6.1
pyelftools==0.24
Pygments==2.2.0
pymongo==3.5.0
python-magic==0.4.13
pytz==2017.2
rarfile==3.0
requests==2.18.4
rfc6266==0.0.4
six==1.10.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.6.3
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
sphinxcontrib-websupport==1.0.1
typing==3.6.2
urllib3==1.22
vine==1.1.4
volatility==2.6
Werkzeug==0.12.2
yara-python==3.6.3
zxcvbn==1.0

########## MongoDB ##########

Version: 3.4.7
Authorization check: True

########## Configuration ##########

Traceback (most recent call last):
File "utils/troubleshoot.py", line 83, in
main()
File "utils/troubleshoot.py", line 80, in main
configuration()
File "utils/troubleshoot.py", line 58, in configuration
for config in Config.find():
File "/home/fame/fame/fame/common/mongo_dict.py", line 24, in find
for obj in objs:
File "/home/fame/fame/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 1134, in next
if len(self.__data) or self._refresh():
File "/home/fame/fame/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 1057, in _refresh
self.__collation))
File "/home/fame/fame/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 949, in __send_message
helpers._check_command_response(doc['data'][0])
File "/home/fame/fame/env/local/lib/python2.7/site-packages/pymongo/helpers.py", line 210, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: not authorized on fame to execute command { find: "settings", filter: {} }

[Help Wanted] Can't get login page

Description

After installing and configuring, cannot find the Login page. This is installed in an Ubuntu 16.04 LTS vm that is being used as an offline analysis environment. The documentation doesn't state where the login page will be, so I have used the vm's IP address

Steps to Reproduce

Do clean install
utils/run.sh utils/install.py
enter the VM's IP when prompted for Domain
let install process finish
visit IP

Expected behavior

Should go to the login page, but no matter what I set the domain to, I don't get a login page

Actual behavior

I wouldn't know, I can't even get the login page

Debug

[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-4.8.0-52-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12


########## DEPENDENCIES ###########

alabaster==0.7.10
amqp==2.1.4
androguard==3.0.1
Babel==2.4.0
bamfdetect==1.6.13
beautifulsoup4==4.6.0
billiard==3.5.0.2
bs4==0.0.1
celery==4.0.2
certifi==2017.4.17
chardet==3.0.4
click==6.7
docutils==0.13.1
Flask==0.12.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.4.6
future==0.16.0
gitdb2==2.0.2
GitPython==2.1.3
googleplay-api==0.1.0
idna==2.5
ijson==2.3
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.6
kombu==4.0.2
LEPL==5.1.3
markdown2==2.3.4
MarkupSafe==1.0
oletools==0.50
pbkdf2==1.3
pefile==2016.3.28
protobuf==3.3.0
pycrypto==2.6.1
pyelftools==0.24
Pygments==2.2.0
pymongo==3.4.0
python-magic==0.4.13
pytz==2017.2
rarfile==3.0
requests==2.17.3
-e git+https://github.com/g2p/rfc6266@cad58963ed13f5e1068fcc9e4326123b6b2bdcf8#egg=rfc6266
six==1.10.0
smmap2==2.0.2
snowballstemmer==1.2.1
Sphinx==1.6.2
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
sphinxcontrib-websupport==1.0.1
typing==3.6.1
urllib3==1.21.1
vine==1.1.3
volatility==2.6
Werkzeug==0.12.2
yara-python==3.6.1
zxcvbn==1.0


########## MongoDB ##########

Version: 3.4.4
Authorization check: True

########## Configuration ##########

types: True
virustotal: False
email: False
malware_config: False
volatility: True

Modules:

McAfee                    Antivirus            Disabled   Configured     
Sophos                    Antivirus            Disabled   Configured     
Symantec                  Antivirus            Disabled   Not Configured 
apk                       Processing           Disabled   Configured     
apk_verification          Processing           Disabled   Not Configured 
bamfdetect                Processing           Disabled   Configured     
cuckoo_modified           Processing           Disabled   Configured     
cutthecrap                Processing           Disabled   Not Configured 
eml                       Processing           Disabled   Configured     
joe                       Processing           Disabled   Not Configured 
marcher_config            Processing           Disabled   Configured     
mem_yara                  Processing           Disabled   Not Configured 
office_macros             Processing           Disabled   Configured     
pdf                       Processing           Disabled   Configured     
url_download              Processing           Disabled   Configured     
zip                       Processing           Disabled   Configured     
slack                     Reporting            Disabled   Not Configured 
Yeti                      Threat Intelligence  Disabled   Not Configured 
virtualbox                Virtualization       Disabled   Configured

Delete submited samples

Please how do i delete the samples already submitted and lastly a sample i have submited says #PENDING and it been long

PyPi package `zxcvbn` taken over by `zxcvbn-python`

Description

PyPi package zxcvbn taken over by zxcvbn-python (see discussion here).

Pinning zxcvbn==1.0 in requirements.txt solves the issue.

Steps to Reproduce

As of 11 Apr 2018,

$ git clone https://github.com/certsocietegenerale/fame
$ cd fame
$ utils/run.sh utils/install.py
$ utils/run.sh webserver.py

Expected behavior

FAME webserver should start.

Actual behavior

(env) fame@fame ~/fame $ utils/run.sh webserver.py 
[+] Using existing virtualenv.

Traceback (most recent call last):
  File "webserver.py", line 20, in <module>
    from web.views.users import UsersView
  File "/home/fame/fame/web/views/users.py", line 14, in <module>
    auth_module = import_module('web.auth.{}.views'.format(fame_config.auth))
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/fame/fame/web/auth/user_password/views.py", line 3, in <module>
    from zxcvbn import password_strength
ImportError: cannot import name password_strength

Debug

(env) fame@fame ~/fame $ utils/run.sh utils/troubleshoot.py 
[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-4.10.0-38-generic-x86_64-with-LinuxMint-18.3-sylvia
Python: 2.7.12


########## DEPENDENCIES ###########

alabaster==0.7.10
amqp==2.2.2
Babel==2.5.3
billiard==3.5.0.3
celery==4.1.0
certifi==2018.1.18
chardet==3.0.4
click==6.7
docutils==0.14
Flask==0.12.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
gitdb2==2.0.3
GitPython==2.1.9
idna==2.6
imagesize==1.0.0
itsdangerous==0.24
Jinja2==2.10
kombu==4.1.0
LEPL==5.1.3
markdown2==2.3.5
MarkupSafe==1.0
packaging==17.1
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-magic==0.4.15
pytz==2018.4
requests==2.18.4
rfc6266==0.0.4
six==1.11.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.7.2
sphinx-rtd-theme==0.3.0
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.0.1
typing==3.6.4
urllib3==1.22
vine==1.1.4
Werkzeug==0.14.1
zxcvbn==4.4.25


########## MongoDB ##########

Version: 3.6.4
Authorization check: True

########## Configuration ##########

types: True
virustotal: False
email: False
malware_config: False
volatility: True

Modules:

McAfee                    Antivirus            Disabled   Configured     
Sophos                    Antivirus            Disabled   Configured     
Symantec                  Antivirus            Disabled   Not Configured 
apk                       Processing           Disabled   Configured     
apk_verification          Processing           Disabled   Not Configured 
bamfdetect                Processing           Disabled   Configured     
cuckoo                    Processing           Disabled   Configured     
cuckoo_modified           Processing           Disabled   Configured     
cutthecrap                Processing           Disabled   Not Configured 
eml                       Processing           Disabled   Configured     
joe                       Processing           Disabled   Not Configured 
marcher_config            Processing           Disabled   Configured     
mem_yara                  Processing           Disabled   Not Configured 
office_macros             Processing           Disabled   Configured     
pdf                       Processing           Disabled   Configured     
rat_decoders              Processing           Disabled   Configured     
url_download              Processing           Disabled   Configured     
zip                       Processing           Disabled   Configured     
slack                     Reporting            Disabled   Not Configured 
Yeti                      Threat Intelligence  Disabled   Not Configured 
kvm                       Virtualization       Disabled   Configured     
virtualbox                Virtualization       Disabled   Configured

How to debug "Could not send to Yeti" errors

I'm looking for suggestions on how to best debug "Could not send to Yeti" errors for Fame observables. I am getting a "200" status code on the Yeti side, but no other indications of errors.

Hints would be appreciated.

fame installation failed or not running

Description

[i tried fame installation by following tutorial, but i do not know when i want to try to access fame did not work, can anyone help me?
thank you]

Steps to Reproduce

[Describe the steps to reproduce]

Expected behavior

[How are you expecting the application to behave?]

Actual behavior

[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]

Debug

[Include the output of utils/run.sh utils/troubleshoot.py]

Incompatibility with pip>=10

Description

FAME cannot be installed when using latest pip versions.

Steps to Reproduce

Follow the installation manual.

Expected behavior

Working installation.

Actual behavior

~/fame$ utils/run.sh utils/install.py
[+] Using existing virtualenv.

[+] Installing requirements ...
~/fame$

Additional info

This is because pip.main no longer exists, it is now pip._internal.main. This technique is NOT used only in the install.

Missing dependency

Description

Got this error in configuration tab for AntiVirus

python /home/USER/fame/fame/modules/community/antivirus/mail/install.py: error on 'fame':

Missing dependency: 7z

Looks like a requirement is need. (Running on fresh install of Ubuntu 16.04)

Please add this package for install:
p7zip-full

Best regards,

How to compile fame on a Raspberry Pi 3B?

Hello I like to have fame on my Raspberry Pi 3B but since mongodb is only version v2.4.14 I can't rely on your documentation. Can you explain me how to correctly make the admin user and configuring the databases?

Greetings,

SquadraMunter

Default populate malware_config

Would it be possible to provide some default values for this configuration ?

Office_macro processor error on file with no extension.

In testing FAME when unzipped files do not have an extension attached, just the MD5 as the filename. FAME does detect the correct file type through strings or the file command and reports that data in the UI. But, the office_macros module errors on a word document. I'm assuming the error is due to no file extension attached to the file. The file does not have macros, so I guess I would expect the results to indicate that.

Example file: https://www.virustotal.com/en/file/ef9495d4d279e595083a92d044561da1f7c48f8885e98eab2bc745e48cb93028/analysis/, but when you download it, rename to the md5hash with no extension.

Steps to Reproduce
Upload a word document file with no extension inside a zip file.

Expected behavior
Either a message is returned that the document does not contain macros, or appending the extension to the file based on other static data information at processing time.

Actual behavior
The office_macros module errors out and I do not get the macros for further analysis.

Debug
2017-03-28 10:25: debug: Trying to run office_macros
2017-03-28 10:25: error: office_macros: Could not run on /home/disdude/fame/storage/<storage_loc>/a4aac4740b67cdf90f1068353376d28d.
Traceback (most recent call last):
File "/home/disdude/fame/fame/core/module.py", line 471, in _try_each
return self.each_with_type(target, file_type)
File "/home/disdude/fame/fame/core/module.py", line 430, in each_with_type
return self.each(target)
File "/home/disdude/fame/fame/modules/community/processing/office_macros/office_macros.py", line 45, in each
analysis = sorted(analysis, key=lambda type_decoded_encoded: len(type_decoded_encoded[2]), reverse=True)
TypeError: 'NoneType' object is not iterable

2017-03-28 10:25: debug: Done with office_macros

Install process does not enforce properly formed email, but http ui does.

Description

During initial install process ($ utils/run.sh utils/install.py), administrative user can be created with a malformed email address. But upon use in HTTP UI, validations are made and prevent the use of malformed address.

Steps to Reproduce

Do clean install
utils/run.sh utils/install.py
enter "something" when prompted for Email Address: Do not use @somewhere.tld
let install process finish
visit :4200, try to use account "something"

Client side validation can be bypassed by removing "input type=email", and authentication can proceed as expected.

<input type="email" name="email" placeholder="Enter email" class="form-control">

Expected behavior

Expected initial account creation to throw warning.

Actual behavior

No warning is thrown.

...
[+] Creating first user (as administrator) ...
[?] Full Name: firstuser
[?] Email Address: firstuser
[?] Groups (comma-separated) [cert]: 
[?] Password: 
[?] Confirm: 
[+] User created.
...

Debug

nothing to add.

Hyperlinks to PCAP files

This is more of a feature request, but would it be possible to add hyperlinks to the PCAP files that are created during the analysis?

TemplateNotFound: analysis/index.html

After loging into fame.. i get the error that : TemplateNotFound: analysis/index.html

Feature request: proxy settings configuration

Description

Setting this up in a pretty contained environment with a need to go through the corporate proxy for outbound requests. I'm not seeing a configuration option for this and have been unsuccessful trying to get this to work. Would it be possible to add this option?

Remote workers and nginx

Thanks for making FAME available to the community.

Description

Issue1: when trying to run the PROD version of the FE with nginx it outputs a 500.

nginx_access.logx.x.x.x- - [27/Mar/2017:19:44:52 +0200] "GET / HTTP/1.1" 500 32 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0"
nginx_debug.log

2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "SERVER_NAME: "
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_HOST: hostname:port"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_USER_AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_ACCEPT_LANGUAGE: en-US,en;q=0.5"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_ACCEPT_ENCODING: gzip, deflate"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_COOKIE: session=.COOKIE_STRING"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_DNT: 1"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_CONNECTION: keep-alive"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_UPGRADE_INSECURE_REQUESTS: 1"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http cleanup add: 0000561043AC8698
2017/03/27 19:52:52 [debug] 2302#2302: *1 get rr peer, try: 1
2017/03/27 19:52:52 [debug] 2302#2302: *1 stream socket 9
2017/03/27 19:52:52 [debug] 2302#2302: *1 epoll add connection: fd:9 ev:80002005
2017/03/27 19:52:52 [debug] 2302#2302: *1 connect to unix:///tmp/fame.sock, fd:9 #2
2017/03/27 19:52:52 [debug] 2302#2302: *1 connected
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream connect: 0
2017/03/27 19:52:52 [debug] 2302#2302: *1 posix_memalign: 0000561043AA8E40:128 @16
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream send request
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream send request body
2017/03/27 19:52:52 [debug] 2302#2302: *1 chain writer buf fl:0 s:978
2017/03/27 19:52:52 [debug] 2302#2302: *1 chain writer in: 0000561043AC86D0
2017/03/27 19:52:52 [debug] 2302#2302: *1 writev: 978 of 978
2017/03/27 19:52:52 [debug] 2302#2302: *1 chain writer out: 0000000000000000
2017/03/27 19:52:52 [debug] 2302#2302: *1 event timer add: 9: 60000:1490637232929
2017/03/27 19:52:52 [debug] 2302#2302: *1 http finalize request: -4, "/?" a:1, c:2
2017/03/27 19:52:52 [debug] 2302#2302: *1 http request count:2 blk:0
2017/03/27 19:52:52 [debug] 2302#2302: *1 post event 0000561043AEF320
2017/03/27 19:52:52 [debug] 2302#2302: *1 post event 0000561043AEF380
2017/03/27 19:52:52 [debug] 2302#2302: *1 delete posted event 0000561043AEF320
2017/03/27 19:52:52 [debug] 2302#2302: *1 http run request: "/?"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream check client, write event:1, "/"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream recv(): -1 (11: Resource temporarily unavailable)
2017/03/27 19:52:52 [debug] 2302#2302: *1 delete posted event 0000561043AEF380
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream request: "/?"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream dummy handler
2017/03/27 19:52:52 [debug] 2302#2302: *1 post event 0000561043AEF380
2017/03/27 19:52:52 [debug] 2302#2302: *1 delete posted event 0000561043AEF380
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream request: "/?"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream dummy handler
2017/03/27 19:52:52 [debug] 2302#2302: *1 post event 0000561043ADD370
2017/03/27 19:52:52 [debug] 2302#2302: *1 post event 0000561043AEF380
2017/03/27 19:52:52 [debug] 2302#2302: *1 delete posted event 0000561043ADD370
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream request: "/?"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream process header
2017/03/27 19:52:52 [debug] 2302#2302: *1 malloc: 0000561043AAD3F0:4096
2017/03/27 19:52:52 [debug] 2302#2302: *1 recv: fd:9 104 of 4096
2017/03/27 19:52:52 [debug] 2302#2302: *1 http uwsgi status 500 "500 Internal Server Error"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http uwsgi header: "Connection: close"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http uwsgi header: "Content-Type: text/plain"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http uwsgi header done
2017/03/27 19:52:52 [debug] 2302#2302: *1 xslt filter header
2017/03/27 19:52:52 [debug] 2302#2302: *1 HTTP/1.1 500 Internal Server Error
Server: nginx/1.10.0 (Ubuntu)
Date: Mon, 27 Mar 2017 17:52:52 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive

Issue2: when trying to run a remote worker (this time with DEV version because I can't run PROD) I get this on the worker: 500 Server Error: INTERNAL SERVER ERROR for url: http://172.16.20.10:4200/modules/download I also tried to point the connection to the extranet address I gave to this host but didn't work.

And this on the server:

172.16.20.50 - - [27/Mar/2017 20:01:08] "GET /modules/download HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1994, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app
    response = self.handle_exception(e)
  File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask_classy.py", line 200, in proxy
    response = view(**request.view_args)
  File "/home/gg/git_code/fame/web/views/helpers.py", line 109, in inner
    return func(*args, **kwargs)
  File "/home/gg/git_code/fame/web/views/modules.py", line 448, in download
    f, path = mkstemp(dir=fame_config.temp_path)
  File "/usr/lib/python2.7/tempfile.py", line 314, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib/python2.7/tempfile.py", line 244, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
OSError: [Errno 2] No such file or directory: '/home/gg/git_code/fame/temp/tmpn3cyU8'

Steps to Reproduce

I have installed FAME as described on the Documentation web page.
I am running a dedicated instance FE for the WEB server, a dedicated instance DB and trying to run a dedicated remote worker instance.
The only step not done from the Documentation is the DB auth (is this critical? I hope this is not the issue.)

Debug

[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-4.4.0-66-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12


########## DEPENDENCIES ###########

alabaster==0.7.10
amqp==2.1.4
appdirs==1.4.3
Babel==2.4.0
billiard==3.5.0.2
celery==4.0.2
click==6.7
docutils==0.13.1
Flask==0.12
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.4.5
gitdb2==2.0.0
GitPython==2.1.3
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.5
kombu==4.0.2
markdown2==2.3.3
MarkupSafe==1.0
packaging==16.8
Pygments==2.2.0
pymongo==3.4.0
pyparsing==2.2.0
python-magic==0.4.13
pytz==2016.10
requests==2.13.0
six==1.10.0
smmap2==2.0.1
snowballstemmer==1.2.1
Sphinx==1.5.3
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
vine==1.1.3
Werkzeug==0.12.1
zxcvbn==1.0


########## MongoDB ##########

Version: 3.4.2
Authorization check: True

########## Configuration ##########

types: True
virustotal: False
email: False
malware_config: False
volatility: True

Modules:

McAfee                    Antivirus            Disabled   Configured     
Sophos                    Antivirus            Disabled   Configured     
Symantec                  Antivirus            Disabled   Not Configured 
apk                       Processing           Enabled    Configured     
apk_verification          Processing           Disabled   Not Configured 
bamfdetect                Processing           Enabled    Configured     
cuckoo_modified           Processing           Disabled   Configured     
eml                       Processing           Enabled    Configured     
joe                       Processing           Disabled   Not Configured 
marcher_config            Processing           Disabled   Configured     
mem_yara                  Processing           Enabled    Configured     
office_macros             Processing           Enabled    Configured     
pdf                       Processing           Enabled    Configured     
url_download              Processing           Enabled    Configured     
zip                       Processing           Enabled    Configured     
slack                     Reporting            Disabled   Not Configured 
Yeti                      Threat Intelligence  Disabled   Not Configured

Could you please help me?

Thanks!
regards.
Guido.

module configuration

Description

[when i try to configure module i told to enter identity, "Triggered By" to fill in what, and if possible ask for an example ]

Steps to Reproduce

[Describe the steps to reproduce]

Expected behavior

[can running module]

Actual behavior

[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]

Debug

[[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-4.15.0-29-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12

########## DEPENDENCIES ###########

alabaster==0.7.10
amqp==2.3.1
androguard==3.2.1
asn1crypto==0.24.0
Babel==2.6.0
backports.functools-lru-cache==1.5
backports.shutil-get-terminal-size==1.0.0
beautifulsoup4==4.6.0
billiard==3.5.0.3
bs4==0.0.1
celery==4.1.1
certifi==2018.4.16
chardet==3.0.4
click==6.7
colorama==0.3.9
cycler==0.10.0
decorator==4.3.0
docutils==0.14
enum34==1.1.6
Flask==1.0.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
future==0.16.0
gitdb2==2.0.3
GitPython==2.1.10
googleplay-api==0.1.0
idna==2.6
ijson==2.3
imagesize==1.0.0
ipython==5.7.0
ipython-genutils==0.2.0
itsdangerous==0.24
Jinja2==2.10
kiwisolver==1.0.1
kombu==4.2.0
LEPL==5.1.3
lxml==4.2.3
markdown2==2.3.5
MarkupSafe==1.0
matplotlib==2.2.2
networkx==2.1
numpy==1.15.0
oletools==0.53.1
packaging==17.1
pathlib2==2.3.2
pbkdf2==1.3
pefile==2017.11.5
pexpect==4.6.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
protobuf==3.6.0
ptyprocess==0.6.0
pycrypto==2.6.1
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-dateutil==2.7.3
python-magic==0.4.15
pytz==2018.4
rarfile==3.0
requests==2.18.4
rfc6266==0.0.4
scandir==1.7
simplegeneric==0.8.1
six==1.11.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.7.5
sphinx-rtd-theme==0.3.1
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.0.1
subprocess32==3.5.2
traitlets==4.3.2
typing==3.6.4
urllib3==1.22
vine==1.1.4
virtualenv==16.0.0
volatility==2.6
wcwidth==0.1.7
Werkzeug==0.14.1
yara-python==3.7.0
zxcvbn==1.0

########## MongoDB ##########

Version: 3.6.4
Authorization check: True

########## Configuration ##########

types: True
virustotal: True
email: False
malware_config: False
volatility: True

Modules:

McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Enabled Configured
apk_verification Processing Enabled Configured
bamfdetect Processing Disabled Configured
cuckoo Processing Enabled Configured
cuckoo_modified Processing Enabled Configured
cutthecrap Processing Disabled Not Configured
eml Processing Enabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Disabled Not Configured
office_macros Processing Enabled Configured
pdf Processing Enabled Configured
rat_decoders Processing Enabled Configured
url_download Processing Enabled Configured
zip Processing Enabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
kvm Virtualization Disabled Configured
virtualbox Virtualization Disabled Configured ]

Adding context to Yeti

Hello,

Is it possible to add a context when we send IOCs to Yeti ?

Best regards,

Internal Certificate Authority for Modules

Description

We use Yeti and FAME with TLS certificates from an internal Certificate Authority. When using the Yeti Threat Intelligence Module, the Yeti module fails because Python requests cannot verify the certificate. Short of not verifying TLS connections, is there a workaround where I can add my CA certificate to FAME?

Steps to Reproduce

Use a self-signed or internal CA-signed Yeti connection within FAME.

Expected behavior

The Yeti Module should work as expected

Actual behavior

I am getting the following error message:

2017-05-09 19:11: error: error in threat intelligence module 'Yeti': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

Delete users

Feature Request

Capability to delete users from the fame portal.

new module installation

Description

cant import new self created processing module to FAME
when trying to run "utils/run.sh utils/single_module.py" i get:
[+] Using existing virtualenv.

[+] Enabling test mode.

/!\ Could not find module 'XXX'

Steps to Reproduce

i saved the code in "/fame/fame/modules/community/processing" under new folder with the module name
also tried creating new module folder under "/fame/fame/modules/processing"
both times i tried "reload" on "Module Repositories" in configuration page

Expected behavior

as i understand a new module should be available in configuration under "processing" in configuration page with the new module name

Actual behavior

no new module shown under "processing" in configuration page

Debug

[Include the output of utils/run.sh utils/troubleshoot.py]
[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-4.4.0-21-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12

########## DEPENDENCIES ###########

alabaster==0.7.11
amqp==2.3.2
androguard==3.2.1
asn1crypto==0.24.0
Babel==2.6.0
backports.functools-lru-cache==1.5
backports.shutil-get-terminal-size==1.0.0
bamfdetect==1.6.13
beautifulsoup4==4.6.3
billiard==3.5.0.4
bs4==0.0.1
celery==4.1.1
certifi==2018.8.24
chardet==3.0.4
click==6.7
colorama==0.3.9
cycler==0.10.0
decorator==4.3.0
docutils==0.14
enum34==1.1.6
Flask==1.0.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
future==0.16.0
gitdb2==2.0.4
GitPython==2.1.11
googleplay-api==0.1.0
idna==2.6
ijson==2.3
imagesize==1.0.0
ipython==5.8.0
ipython-genutils==0.2.0
itsdangerous==0.24
Jinja2==2.10
kiwisolver==1.0.1
kombu==4.2.1
LEPL==5.1.3
lxml==4.2.4
markdown2==2.3.5
MarkupSafe==1.0
matplotlib==2.2.3
networkx==2.1
numpy==1.15.1
oletools==0.53.1
packaging==17.1
pathlib2==2.3.2
pbkdf2==1.3
pefile==2018.8.8
pexpect==4.6.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
protobuf==3.6.1
ptyprocess==0.6.0
pycrypto==2.6.1
pyelftools==0.25
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-dateutil==2.7.3
python-magic==0.4.15
pytz==2018.5
rarfile==3.0
requests==2.18.4
rfc6266==0.0.4
scandir==1.9.0
simplegeneric==0.8.1
six==1.11.0
smmap2==2.0.4
snowballstemmer==1.2.1
Sphinx==1.7.8
sphinx-rtd-theme==0.3.1
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.1.0
subprocess32==3.5.2
traitlets==4.3.2
typing==3.6.6
urllib3==1.22
vine==1.1.4
volatility==2.6
wcwidth==0.1.7
Werkzeug==0.14.1
yara-python==3.8.1
zxcvbn==1.0

########## MongoDB ##########

Version: 4.0.2
Authorization check: True

########## Configuration ##########

types: True
virustotal: True
email: False
malware_config: False
volatility: True

Modules:

McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Enabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Enabled Configured
cuckoo Processing Disabled Configured
cuckoo_modified Processing Disabled Configured
cutthecrap Processing Disabled Not Configured
eml Processing Enabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Enabled Configured
office_macros Processing Enabled Configured
pdf Processing Enabled Configured
rat_decoders Processing Enabled Configured
url_download Processing Disabled Configured
zip Processing Enabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
kvm Virtualization Disabled Configured
virtualbox Virtualization Disabled Configured

[+] Generating SSH key ... ImportError: No module named rfc6266

Description

[+] Generating SSH key ... ImportError: No module named rfc6266
During fresh Installation of fame I've encountered the following error:
[+] Generating SSH key ...
Traceback (most recent call last):
File "utils/install.py", line 227, in
main()
File "utils/install.py", line 221, in main
perform_local_installation(context)
File "utils/install.py", line 143, in perform_local_installation
from fame.core import fame_init
File "/home/lacerator/fame/fame/core/init.py", line 2, in
from fame.core.module_dispatcher import dispatcher
File "/home/lacerator/fame/fame/core/module_dispatcher.py", line 8, in
from fame.common.utils import get_class, iterify, unique_for_key
File "/home/lacerator/fame/fame/common/utils.py", line 10, in
from rfc6266 import parse_requests_response
ImportError: No module named rfc6266

I have sudo pip install rfc6266 and still the same issue happens.

Steps to Reproduce

I'm running inside a VM using VMware workstation 12

Install Ubuntu 16.04 LTS latest updates and build
Install MongoDB shell version v3.4.5
Install Fame

Expected behavior

[How are you expecting the application to behave?]
Fame to install without errors.

Actual behavior

[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]
[+] Generating SSH key ...
Traceback (most recent call last):
File "utils/install.py", line 227, in
main()
File "utils/install.py", line 221, in main
perform_local_installation(context)
File "utils/install.py", line 143, in perform_local_installation
from fame.core import fame_init
File "/home/lacerator/fame/fame/core/init.py", line 2, in
from fame.core.module_dispatcher import dispatcher
File "/home/lacerator/fame/fame/core/module_dispatcher.py", line 8, in
from fame.common.utils import get_class, iterify, unique_for_key
File "/home/lacerator/fame/fame/common/utils.py", line 10, in
from rfc6266 import parse_requests_response
ImportError: No module named rfc6266

Debug

[Include the output of utils/run.sh utils/troubleshoot.py]

utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-4.4.0-66-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12

########## DEPENDENCIES ###########

alabaster==0.7.10
amqp==2.1.4
Babel==2.4.0
billiard==3.5.0.2
celery==4.0.2
certifi==2017.4.17
chardet==3.0.4
click==6.7
docutils==0.13.1
Flask==0.12.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.0
gitdb2==2.0.2
GitPython==2.1.5
idna==2.5
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.6
kombu==4.0.2
LEPL==5.1.3
markdown2==2.3.4
MarkupSafe==1.0
Pygments==2.2.0
pymongo==3.4.0
python-magic==0.4.13
pytz==2017.2
requests==2.18.1
-e git+https://github.com/g2p/rfc6266@cad58963ed13f5e1068fcc9e4326123b6b2bdcf8#egg=rfc6266
six==1.10.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.6.2
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
sphinxcontrib-websupport==1.0.1
typing==3.6.1
urllib3==1.21.1
vine==1.1.3
Werkzeug==0.12.2
zxcvbn==1.0

########## MongoDB ##########

Version: 3.4.5
Authorization check: True

########## Configuration ##########

Modules:

Getting what appears to be a timeout with Cuckoo

I have verified that Cuckoo was able to download and analyze the file so I'm looking for suggestions as to how to proceed. The parameters for the Cuckoo module are:

WAIT_TIMEOUT 5400
WAIT_STEP 30
ANALYSIS_TIME 300

2018-10-16 08:52: debug: Trying to run cuckoo
2018-10-16 09:05: error: cuckoo: Could not run on http://microsoftupdate.dynamicdns.org.uk/host/290.exe.
Traceback (most recent call last):
File "/home/cirt/fame/fame/core/module.py", line 492, in _try_each
return self.each_with_type(target, file_type)
File "/home/cirt/fame/fame/modules/community/processing/cuckoo/cuckoo.py", line 97, in each_with_type
self.process_report()
File "/home/cirt/fame/fame/modules/community/processing/cuckoo/cuckoo.py", line 159, in process_report
self.extract_info(response)
File "/home/cirt/fame/fame/modules/community/processing/cuckoo/cuckoo.py", line 166, in extract_info
for prefix, event, value in parser:
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/common.py", line 65, in parse
for event, value in basic_events:
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 185, in basic_parse
for value in parse_value(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 119, in parse_value
for event in parse_object(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 163, in parse_object
for event in parse_value(lexer, None, pos):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 119, in parse_value
for event in parse_object(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 163, in parse_object
for event in parse_value(lexer, None, pos):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 116, in parse_value
for event in parse_array(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 138, in parse_array
for event in parse_value(lexer, symbol, pos):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 119, in parse_value
for event in parse_object(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 163, in parse_object
for event in parse_value(lexer, None, pos):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 116, in parse_value
for event in parse_array(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 138, in parse_array
for event in parse_value(lexer, symbol, pos):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 119, in parse_value
for event in parse_object(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 163, in parse_object
for event in parse_value(lexer, None, pos):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 119, in parse_value
for event in parse_object(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 170, in parse_object
pos, symbol = next(lexer)
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 65, in Lexer
data = f.read(buf_size)
File "/home/cirt/fame/env/lib/python2.7/codecs.py", line 488, in read
newdata = self.stream.read(size)
File "/usr/lib/python2.7/socket.py", line 384, in read
data = self._sock.recv(left)
error: [Errno 104] Connection reset by peer

creating class to connect module to fame

Description

[failed to connect the module with fame, confused in placing the creation class for the module]

Steps to Reproduce

[I tried to create a python class and sample documentation like on the web, but I am still confused in placing a class, so that it becomes inaccessible.]

Expected behavior

[I hope the module and fame can connect and I can use the features that are fame as well as possible.
]

Actual behavior

[fame not running]

Debug

[root@bismillah-VirtualBox:~/fame# utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-4.10.0-28-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12

########## DEPENDENCIES ###########

alabaster==0.7.10
amqp==2.3.1
androguard==3.2.1
asn1crypto==0.24.0
Babel==2.6.0
backports.functools-lru-cache==1.5
backports.shutil-get-terminal-size==1.0.0
beautifulsoup4==4.6.0
billiard==3.5.0.3
bs4==0.0.1
celery==4.1.1
certifi==2018.4.16
chardet==3.0.4
click==6.7
colorama==0.3.9
cycler==0.10.0
decorator==4.3.0
docutils==0.14
enum34==1.1.6
Flask==1.0.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
future==0.16.0
gitdb2==2.0.3
GitPython==2.1.10
googleplay-api==0.1.0
idna==2.6
ijson==2.3
imagesize==1.0.0
ipython==5.7.0
ipython-genutils==0.2.0
itsdangerous==0.24
Jinja2==2.10
kiwisolver==1.0.1
kombu==4.2.0
LEPL==5.1.3
lxml==4.2.3
markdown2==2.3.5
MarkupSafe==1.0
matplotlib==2.2.2
networkx==2.1
numpy==1.15.0
oletools==0.53.1
packaging==17.1
pathlib2==2.3.2
pbkdf2==1.3
pefile==2017.11.5
pexpect==4.6.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
protobuf==3.6.0
ptyprocess==0.6.0
pycrypto==2.6.1
pyelftools==0.24
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-dateutil==2.7.3
python-magic==0.4.15
pytz==2018.4
rarfile==3.0
requests==2.18.4
rfc6266==0.0.4
scandir==1.7
simplegeneric==0.8.1
six==1.11.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.7.5
sphinx-rtd-theme==0.3.1
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.0.1
subprocess32==3.5.2
traitlets==4.3.2
typing==3.6.4
urllib3==1.22
vine==1.1.4
volatility==2.6
wcwidth==0.1.7
Werkzeug==0.14.1
yara-python==3.7.0
zxcvbn==1.0

########## MongoDB ##########

Version: 3.6.4
Authorization check: True

########## Configuration ##########

types: True
virustotal: True
email: False
malware_config: False
volatility: True

Modules:

McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Enabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Disabled Configured
cuckoo Processing Enabled Configured
cuckoo_modified Processing Enabled Configured
cutthecrap Processing Disabled Not Configured
eml Processing Enabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Disabled Not Configured
office_macros Processing Enabled Configured
pdf Processing Enabled Configured
rat_decoders Processing Enabled Configured
url_download Processing Enabled Configured
zip Processing Enabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
kvm Virtualization Disabled Configured
virtualbox Virtualization Disabled Configured]

Hash submission to API send html code instead of json

Description

[Please provide a description of the issue encountered]
When submiting a Hash as analysis through API if reulstant hash file is not found FAME returns bit html code instead of json reply.

Steps to Reproduce

[Describe the steps to reproduce]

headers = {'Accept': 'application/json',
 'X-API-KEY': 'my_api_key'}

params2 = {
    'options[allow_internet_access]':'on',
    'options[analysis_time]':"300", 'groups':'*','options[tag]':'Honeypot','hash':'8971fc79d73e2541cf5a27e8ad5e971c'
}

r2 = requests.post(submit_url,data=params2,headers=headers,verify=False)

Expected behavior

[How are you expecting the application to behave?]
I was expecting json reply similar to
u'{"analysis": {"support_files": {}, "logs": ["2017-07-25 11:23: debug: Trying to queue module \'bamfdetect\'", "2017-07-25 11:23: debug: Trying to queue module \'eml\'", "2017-07-25 11:23: debug: Trying to queue module \'office_macros\'", "2017-07-25 11:23: debug: Trying to queue module \'pdf\'", "2017-07-25 11:23: debug: Trying to queue module \'zip\'", "2017-07-25 11:23: debug: Trying to queue module \'fireeye_ax\'", "2017-07-25 11:23: debug: Trying to queue module \'virustotal_report\'", "2017-07-25 11:23: debug: Trying to queue module \'payload_security\'"], "extractions": [], "results": {}, "module": null, "date": {"$date": 1500981796720}, "file": {"$oid": "59772a24e6c7db09969802f5"}, "iocs": [], "executed_modules": [], "probable_names": [], "extracted_files": [], "status": "pending", "tags": [], "groups": ["*"], "pending_modules": ["fireeye_ax", "virustotal_report", "payload_security"], "analyst": {"$oid": "59663083e6c7db099698027a"}, "waiting_modules": [], "canceled_modules": [], "threat_intelligence": {}, "generated_files": {}, "_id": {"$oid": "59772a24e6c7db09969802f6"}, "options": {}}}'

Actual behavior

[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]
API call returned following html

u'<!doctype html>\n\n\n\t\n\t\n\t\n\n\t<title>FAME</title>\n\n\t<meta content='width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0' name='viewport' />\n \n\n\n \n \n\n \n \n\n \n \n\n \n \n\n \n \n <link href='https://fonts.googleapis.com/css?family=Roboto:400,700,300\' rel='stylesheet' type='text/css'>\n \n\n \n \n\n \n <script src="/static/js/jquery-1.11.3.min.js" type="text/javascript"></script>\n\t<script src="/static/js/bootstrap.min.js" type="text/javascript"></script>\n\n \n\t<script src="/static/js/template.js"></script>\n <script src="/static/js/checkbox.js"></script>\n\n \n <script src="/static/js/bootstrap-notify.min.js"></script>\n\n \n <script src="/static/js/fame.js"></script>\n\n \n \n <script src="/static/js/fileinput.min.js"></script>\n\n \n <script src="/static/js/highlight.min.js"></script>\n\n \n <script src="/static/js/tagsinput.js"></script>\n\n \n <script src="/static/js/typeahead.bundle.min.js"></script>\n\n \n <script src="/static/js/handlebars.min.js"></script>\n\n\n\n

\n

\n\n \t

\n

\n \n FAME\n \n

\n\n

\n

\n \n \n
Last Analyses
\n \n
\n \n
\n \n \n \n
finished
\n
\n\n
\n \n 64d742004f2efc696fca74dfca9aaeb8\n \n
\n
\n 64d742004f2efc696fca74dfca9aaeb8.bin\n
\n\n \n \n
\n \n
\n \n \n \n
finished
\n
\n\n
\n \n aa51b318599e9402b5bda6937523105c\n \n
\n
\n x87junkie\n
\n\n \n \n
\n \n
\n \n \n \n
finished
\n
\n\n
\n \n 98a716f8827761c81374d83f1d051312\n \n
\n
\n 98a716f8827761c81374d83f1d051312.bin\n
\n\n \n \n
\n \n
\n \n \n \n
finished
\n
\n\n
\n \n c98bb1b895ff1a13ed270dcb23c5e175\n \n
\n
\n Eviction_Notice.doc\n
\n\n \n \n

\n

\n \t

\n

\n\n

\n \n

\n

\n \n Toggle navigation\n \n \n \n \n \n

\n

\n \n

\n \n \n

\n \n\n

\n \n Configs\n \n
\n \n Objects\n \n
\n \n Analyses\n \n
\n \n Submit\n \n
\n \n \n \n\n
- Profile
- Configuration
- Users
- Logout
\n

\n

\n \n\n\n

\n

\n \n \n \n\n \n

\n\t

\n

Submit a new file

\n

And start an analysis

\n

\n \n

\n

File
URL

\n

\n\n

\n

\n \n

\n

\n \n

\n

\n \n

\n

\n\n

\n \n

\n\n

\n Share with: \n \n

\n\n \n\n

\n Submit\n

\n \n

\n

\n\n\n<script>\n $('.tags-input').tagsInput();\n</script>\n\n\n<script id="modules-results" type="text/x-handlebars-template">\n

\n

\n</script>\n\n<script id="modules-empty" type="text/x-handlebars-template">\n

\n No matching module found.\n

\n</script>\n\n<script src="/static/js/modules-typeahead.js"></script>\n\n\n

\n

\n\n

\n

\n\n\n \n <script>\n \n\n \n \n var category = "danger";\n \n\n $.notify({\n message: "No file found with this hash."\n }, {\n offset: {\n y: 50,\n x: 30\n },\n type: category\n });\n \n \n </script>\n \n\n\n\n'

Debug

[Include the output of utils/run.sh utils/troubleshoot.py]

fame menu issues

Description

[when I managed to access the fame, I got into trouble with the detail analysis menu, where when I tried to do the analysis I only get some menu just like: file details, execution path, logs and dimenu path there is still some constraint, when I read in fame documentation there some menu options such as: execution path, observables, extracttions, detailed results, and logs.
roughly for that problem how is the solution? thank you very much.]

Steps to Reproduce

[I hope fame that I install can run according to the documentation and can run well]

Expected behavior

[I only get some menu just like: file details, execution path, logs ]

Actual behavior

[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]

Debug

[Include the output of utils/run.sh utils/troubleshoot.py]

certsocietegenerale / fame Goto Github PK

fame's Introduction

What is FAME ?

Installation

Community

Technical Specs

Credits

fame's People

Contributors

Stargazers

Watchers

Forkers

fame's Issues

Description

Steps to Reproduce

Expected behavior

Actual behavior

Debug

Description

Steps to Reproduce

Expected behavior

Actual behavior

Debug

Description

Steps to Reproduce

Expected behavior

Actual behavior

Debug

Description

Steps to Reproduce

Expected behavior

Actual behavior

Debug

Description

Description

Steps to Reproduce

Expected behavior

Actual behavior

Debug

Description

Steps to Reproduce

Expected behavior

Actual behavior

Debug

Description

Steps to Reproduce

Description

Steps to Reproduce

Expected behavior

Actual behavior

Debug

Description

Steps to Reproduce

Expected behavior

Actual behavior

Debug

Description

Steps to Reproduce

Expected behavior

Actual behavior

Debug

Description

Steps to Reproduce

Expected behavior

Actual behavior

Debug

Description

Steps to Reproduce

Expected behavior

Actual behavior

Debug

Description

Questions

Description

Debug

Description

Steps to Reproduce

Expected behavior

Actual behavior

Debug