misp / pymisp

Python library using the MISP Rest API

IMPORTANT NOTE: This library will require at least Python 3.10 starting the 1st of January 2024. If you have legacy versions of python, please use the latest PyMISP version that will be released in December 2023, and consider updating your system(s). Anything released within the last 2 years will do, starting with Ubuntu 22.04.

PyMISP - Python Library to access MISP

PyMISP is a Python library to access MISP platforms via their REST API.

PyMISP allows you to fetch events, add or update events/attributes, add or update samples or search for attributes.

Install from pip

It is strongly recommended to use a virtual environment

If you want to know more about virtual environments, python has you covered

Only basic dependencies:

pip3 install pymisp

And there are a few optional dependencies:

• fileobjects: to create PE/ELF/Mach-o objects
• openioc: to import files in OpenIOC format (not really maintained)
• virustotal: to query VirusTotal and generate the appropriate objects
• docs: to generate te documentation
• pdfexport: to generate PDF reports out of MISP events
• url: to generate URL objects out of URLs with Pyfaup
• email: to generate MISP Email objects
• brotli: to use the brotli compression when interacting with a MISP instance

Example:

pip3 install pymisp[virustotal,email]

Install the latest version from repo from development purposes

Note: poetry is required; e.g., "pip3 install poetry"

git clone https://github.com/MISP/PyMISP.git && cd PyMISP
git submodule update --init
poetry install -E fileobjects -E openioc -E virustotal -E docs -E pdfexport -E email

Running the tests

poetry run pytest --cov=pymisp tests/test_*.py

If you have a MISP instance to test against, you can also run the live ones:

Note: You need to update the key in tests/testlive_comprehensive.py to the automation key of your admin account.

poetry run pytest --cov=pymisp tests/testlive_comprehensive.py

Samples and how to use PyMISP

Various examples and samples scripts are in the examples/ directory.

In the examples directory, you will need to change the keys.py.sample to enter your MISP url and API key.

cd examples
cp keys.py.sample keys.py
vim keys.py

The API key of MISP is available in the Automation section of the MISP web interface.

To test if your URL and API keys are correct, you can test with examples/last.py to fetch the events published in the last x amount of time (supported time indicators: days (d), hours (h) and minutes (m)). last.py

cd examples
python3 last.py -l 10h # 10 hours
python3 last.py -l 5d  #  5 days
python3 last.py -l 45m # 45 minutes

Debugging

You have two options here:

1. Pass debug=True to PyMISP and it will enable logging.DEBUG to stderr on the whole module

2. Use the python logging module directly:

import logging
logger = logging.getLogger('pymisp')

# Configure it as you wish, for example, enable DEBUG mode:
logger.setLevel(logging.DEBUG)

Or if you want to write the debug output to a file instead of stderr:

import pymisp
import logging

logger = logging.getLogger('pymisp')
logging.basicConfig(level=logging.DEBUG, filename="debug.log", filemode='w', format=pymisp.FORMAT)

Test cases

1. The content of mispevent.py is tested on every commit
2. The test cases that require a running MISP instance can be run the following way:

# From poetry

pytest --cov=pymisp tests/test_*.py tests/testlive_comprehensive.py:TestComprehensive.[test_name]

Documentation

The documentation is available here.

Jupyter notebook

A series of Jupyter notebooks for PyMISP tutorial are available in the repository.

Everything is a Mutable Mapping

... or at least everything that can be imported/exported from/to a json blob

AbstractMISP is the master class, and inherits from collections.MutableMapping which means the class can be represented as a python dictionary.

The abstraction assumes every property that should not be seen in the dictionary is prepended with a _, or its name is added to the private list __not_jsonable (accessible through update_not_jsonable and set_not_jsonable.

This master class has helpers that make it easy to load, and export to, and from, a json string.

MISPEvent, MISPAttribute, MISPObjectReference, MISPObjectAttribute, and MISPObject are subclasses of AbstractMISP, which mean that they can be handled as python dictionaries.

MISP Objects

Creating a new MISP object generator should be done using a pre-defined template and inherit AbstractMISPObjectGenerator.

Your new MISPObject generator must generate attributes and add them as class properties using add_attribute.

When the object is sent to MISP, all the class properties will be exported to the JSON export.

Installing PyMISP on a machine with no internet access

This is done using poetry and you need to have this repository cloned on your machine. The commands below have to be run from inside the cloned directory.

1. From a machine with access to the internet, get the dependencies:

mkdir offline
poetry export --all-extras  > offline/requirements.txt
poetry run pip download -r offline/requirements.txt -d offline/packages/

2. Prepare the PyMISP Package

poetry build
mv dist/*.whl offline/packages/

3. Copy the content of offline/packages/ to the machine with no internet access.

4. Install the packages:

python -m pip install --no-index --no-deps packages/*.whl

License

PyMISP is distributed under an open source license. A simplified 2-BSD license.

pymisp's Issues

Cannot instantiate basestring (Python2)

An exception is thrown at api.py line 218 for Python2 as the basestring type cannot be instantiated.

Offending line:
errors.append(basestring(response.status_code))

Documentation:
https://docs.python.org/2.7/library/functions.html#basestring

should dateutil module be a hard requirement?

In pymisp/misp_event.py:

 11 try:
 12     from dateutil.parser import parse
 13 except ImportError:
 14     pass

But it is used like it must exists.

345     def set_date(self, date, ignore_invalid=False):
346         if isinstance(date, basestring) or isinstance(date, unicode):
347             self.date = parse(date).date()
348         elif isinstance(date, datetime.datetime):
349             self.date = date.date()
350         elif isinstance(date, datetime.date):
351             self.date = date

I can fix that, but what would you prefer?

• Modify set_date() behavior with a has_dateutil
• Make a successful import mandatory?

Setting published=True on PyMISP.new_event() does not work

After adding attributes to the newly created Event no REST call to update the "published" status is made.

Can't add hashes for filenames with regex

When I try to add a hash for a filename containing a regex like file\d{1}.\d{2}.exe, PyMISP throws this error:

Traceback (most recent call last):
  File "./import.py", line 178, in <module>
    readcsv(sys.argv[1])
  File "./import.py", line 163, in readcsv
    r = mymisp.add_hashes(event, filename=row[1], md5=md5, sha1=sha1, sha256=sha256, comment=comment)
  File "/opt/rh/rh-python34/root/usr/lib/python3.4/site-packages/pymisp/api.py", line 448, in add_hashes
    attributes.append(self._prepare_full_attribute(category, type_value.format('md5'), value.format(md5),
IndexError: tuple index out of range

This is because the filename is interpreted as a format string in https://github.com/MISP/PyMISP/blob/master/pymisp/api.py#L455

Unable to add attributes after creating new event

I'm in the process of trying to convert a large amount of data from a csv into MISP, creating brand new events based on combinations of specific fields (e.g. Reporting_Org+Report_Number = Event description).

When I call misp.new_event(info=event['info']), it creates the new event just fine, but when I try to search it via search('events', **kwargs), it tells me there are no results. I've tried multiple search params (org, from, values), and still am getting a response that it doesn't exist. I have even tried manually creating an event through the web UI, and then searching it, but the API still says it can't find the event I'm looking for.

The strange thing is, once I add an attribute through the web UI and run the exact same search, I get the results I'm looking for. Because of this, I've tried calling add_named_attribute() immediately after creating the new event, but when I pass the new event to add_named_attribute() it says "Invalid event".

Since the MISP API can't seem to find my newly created events until I add an attribute to them, and I apparently can't add an attribute right off the bat through new_event(), what is the best way to go about creating a new event and then immediately adding an attribute to it?

Thanks in advance!

Version: 2.4.34

Code snippet:

def send_to_misp(misp, misp_url, events):
    
    # Loop throuhgh each event
    for event in events:
        # Search for existing event
        kwargs={'values': event['info']
                }
        result = search(misp, misp_url, **kwargs)
        # ^^^ Never finds results, until an attribute is added manually through web UI

        if result['message']=="No matches.":
            #Add new event
            result = misp.new_event(info=event['info'])
            
            # Add Searchable comment
            
            #eid= misp.get_event(result['Event']['id'])
            #print "eid: "+str(eid)
            # Turns out get_event() gives me the same thing that's already in result

            #comment_res = misp.add_named_attribute(result, 'other', result['Event']['info'])            
            comment_res = misp.add_named_attribute(result, 'other', result['Event']['info'], "Other")
            #^^^ Both return 'Invalid event' even though the event exists

attempts to add MISP exports into MISP via python result in empty event

I use the following code to upload either of the attached files and I end up with an empty event in MISP.

MISP
Currently installed version..... v2.4.77 (272d8930944c9e57812e256e4dd72136fef3fd7a)

`import json
from pymisp import PyMISP
from pprint import pprint
misp_url = "http://"
misp_key = "<redacted"
def init(url, key):
return PyMISP(url, key, 'misp')
misp = init(misp_url, misp_key)

#with open('MISP-test-files/vx-misp.xml', 'r') as f:
with open('MISP-test-files/vx-json.json', 'r') as f:
data = json.load(f)
test = misp.add_event(data)

#print(test)
print(json.dumps(data))`

I would prefer to be able to to feed MISP XML directly to the add_event() function but it only accepts json objects. Let me know what other info you require.
Archive.zip

Support for client certificates

We utilize a MISP instance where MISP is protected by SSL/TLS and requires client certificate auth, i.e.

<Directory /var/www/MISP/app/webroot>
    SSLVerifyClient require
    ...

If I'm not mistaken, it is the case that a remote client using PyMISP cannot currently interact with this MISP REST API. Would it be possible to add support for supplying client certificate data in PyMISP to interact with such deployments?

Noteworthy that this seems fairly simplistic using Requests: http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification.

In our use case we'd have various PyMISP clients configured with a user settings file and would add such parameters (key and certificate files, or simply combined key+cert file) in there.

feature request add attribute with a tag

is it possible to add the functionality to add a tag direct with an attribute ?

When you want to do this now, you need to wait for the request to get the uuid to add the tag to the attribute.

Or is there a faster way of adding an attribute and a tag ?

Maybe the "|" should be replaced by the new key?

PyMISP/pymisp/tools/peobject.py

Line 109 in 2bc0745

self.add_attribute('entrypoint-section|position', value='{}|{}'.format(section.name, pos))

PyMISP patch level version strings cause warnings

The PyMISP version string uses major.minor.misp-pl,pymisp-pl which causes the following warning:

The version of PyMISP recommended by the MISP instance (2.4.81) is older than the one you're using now (2.4.81.2). Please upgrade the MISP instance or use an older PyMISP version.

A simple solution would be to strip out the last (4th) pymisp-pl part of the version before comparing.

Adding "indent" param to AbstractMISP.to_json()

I keep getting the error TypeError: Object of type 'MISPObject' is not JSON serializable when running print(json.dumps(misp_object, indent=2)). Looks like there isn't a way to do a json.dumps() call on any MISPObject instances if I want to add extra indentation for prettier printing, I'll need to roll my own method instead (example below). It might be nice to have this functionality baked into the AbstractMISP class's to_json() function with an optional "indent" parameter.

@Rafiot do you think it's worth it to add that in for an admittedly minor use case. If so, I can submit a PR.

Example:

    import copy
    misp_json = copy.deepcopy(misp_object)
    misp_json = misp_json.to_dict()
    for i, attr in enumerate(misp_json["Attribute"]):
        misp_json["Attribute"][i] = attr.to_dict()
    print(json.dumps(misp_json, indent=2))

Proposal:

print(misp_object.to_json(indent=2))

Search for "something.xml" leads to error

When I try a misp.search_index(eventinfo="something.xml") the search fails:

Traceback (most recent call last):
File "api.py", line 130, in
result = misp.search_index(eventinfo=title)
File "/Users/jgo/git/PyMISP/pymisp/api.py", line 814, in search_index
return self._check_response(response)
File "/Users/jgo/git/PyMISP/pymisp/api.py", line 198, in _check_response
raise PyMISPError('Unknown error: {}'.format(response.text))

It seems like MISP is returning XML instead of JSON in case the event title ends with ".xml". I did not expect output formating parameters to be part of the search string itself.

Update event if already exists is not possible any more with add_event

Hi,

I was using until now a pretty old version of PyMISP until now.
The function add_event used to return a requests Response object. This allowed me to get the existing event id when I tried to add an event that already exists (using the Location header).

Now, the add_event return only json body using Response.json() so I'm not able to get the event id in order to update it any more.
What is returned:
{u'url': u'/events', u'message': u'Event already exists, if you would like to edit it, use the url in the location header.', 'errors': [u'Event already exists, if you would like to edit it, use the url in the location header.'], u'name': u'Event already exists, if you would like to edit it, use the url in the location header.'}

Am I doing something wrong ?

Add additional filter options to the feed generator

Such as timeframes (from, to, last)

add_attribute_tag

Can we think about a method called add_attribute_tag, which will be a wrapper for

header will most likely be:

def add_tag(self, attribute_id, tag):
add_tag(attribute_id,tag,attribute=True)

Concerns or other ideas?

New dependencies?

The test suggest that pydeep and python-magic are new dependencies or at least recommended requirements. If they are, these package names should be mentioned in setup.py's install_requires or extra_requires.

And pydeep does not have a stable release, not a release and not on pypi. The latest release 0.2 is from 2013, 4 years ago. That supports up to python 2.6!

search parameter validation

The PyMISP search functionality does not validate the parameter keys that are passed. If you misspell a parameter (attribute_type vs. type_attribute), PyMISP will simply ignore it. This is dangerous insofar as you cannot determine on first glance that something went wrong, since PyMISP returns valid JSON.

My simple-minded approach would be to pop each parameter from **kwargs instead of getting them and alerting the user if arguments from the dict are not popped yet (Unused parameter: attribute_type). Perhaps you have a better idea.

upload_sample always creates new event

The API call upload_sample is broken In PyMISP 2.4.81 with MISP 2.4.81.

The following code is expected to add a sample to an existing event:

import pymisp

url = "https://..."
key = "..." 
misp = pymisp.PyMISP(url, key)

event = misp.new_event(info="Upload Sample does not write into this event") 
eventid = event['Event']['id']
with open('test.txt', 'w') as w:
    w.write('test file')
misp.upload_sample("test", "test.txt", eventid)

Instead of adding the sample to the existing event, a second event is created with title "Malware samples uploaded on 2017-10-23".

PyMISP.search_index() is broken with MISP 2.4.80

Hi,

PyMISP.search_index() is broken when used on a MISP 2.4.80 (it always worked with prior versions). The bug seems to happen with any version of PyMISP (tried with 2.4.71 and after upgrade to 2.4.80).
Here's the message returned by MISP:
The request has been black-holed

Tried manually with curl, it's working.

Provide method to `attachClusterToEvent`

Provide a method to attach galaxies using the API.

Changelog

Is there a changelog somewhere?

It'd be nice to have this as packager/developer to see if there are any changes affecting me.

MISPEvent.add_tag(tag) fails if inital event does not contain tag

If the inital creation via from_dict(**kwargs) does not contain a Tag a call of MISPEvent.add_tag(tag) fails with {AttributeError}'MISPEvent' object has no attribute 'Tag'

misp_get_event(eventid) doesn't return tags for event

I'm using PyMISP to to do some processing on events in MISP

As part of the data I'm processing I would like to have the tags for the event(s), but the json returned from misp_get_event() doesn't have the tags.

I'm using the following code:

misp = PyMISP("https://%s/" % misp_server, misp_key, False, 'json')
source_event = misp.get_event(source_id)

I see that get_stix_event() have an tags parameter to include tags, but since I use json for everything else I would like to avoid rewriting the code to use STIX instead of json

Timestamp and delete=false parameter doesn't work - class MISP Attribute, method set_all_values

Hi,
I'm new MISP admin at NCIRC :)

PyMISP has a bug in set_all_values method in class MISPAttribute (https://github.com/MISP/PyMISP/blob/master/pymisp/mispevent.py#L180). This method doesn’t replicate timestamp and deleted=false value of the attribute. I’ve investigated attribute deletion and it happens because there is condition:
if kwargs.get(‘deleted’):
Which obviously wouldn’t be fullfield if delete kwargs EXISTS but has False value.
Solution:
if kwargs.get(‘deleted’) is not None
Probably this bug exists in some other places but I didn't investigate that.

Is it possible to get the number of sightings for an attribute?

I can see a number of methods available here: https://github.com/MISP/PyMISP/blob/master/pymisp/api.py#L1122

but all appear to add a sighting to an attribute, I can't find any obvious way in PyMISP to get the number of sightings for a given attribute (both positive and false-positive).

If it's not currently possible to do so, could someone just confirm that's the case?

Type entrypt for encrypt

mispevent.py, line 231 (_json method), I assume it should read 'encrypt", not 'entrypt'

Tests: missing mock for attribute deletion

I get this error when running the tests locally (python3 setup.py test outside and inside a build environment):

> python3 setup.py test
running test
Searching for requests-mock
Reading https://pypi.python.org/simple/requests-mock/
Downloading https://pypi.python.org/packages/8d/cb/1267d7294d97e9a3ef24bf1370791da4d2dc6abc0f67626f38f4bf25dfa3/requests-mock-1.3.0.tar.gz#md5=d2efbaf16d19153b7d271628071b4d4b
Best match: requests-mock 1.3.0
Processing requests-mock-1.3.0.tar.gz
Writing /tmp/easy_install-jfyygooo/requests-mock-1.3.0/setup.cfg
Running requests-mock-1.3.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-jfyygooo/requests-mock-1.3.0/egg-dist-tmp-sxdqv22j
creating /home/ele/develop/packaging/devel:languages:python/python-pymisp/pymisp-2.4.81.2/.eggs/requests_mock-1.3.0-py3.6.egg
Extracting requests_mock-1.3.0-py3.6.egg to /home/ele/develop/packaging/devel:languages:python/python-pymisp/pymisp-2.4.81.2/.eggs

Installed /home/ele/develop/packaging/devel:languages:python/python-pymisp/pymisp-2.4.81.2/.eggs/requests_mock-1.3.0-py3.6.egg
Searching for python-magic
Reading https://pypi.python.org/simple/python-magic/
Downloading https://pypi.python.org/packages/65/0b/c6b31f686420420b5a16b24a722fe980724b28d76f65601c9bc324f08d02/python-magic-0.4.13.tar.gz#md5=bbf309c8eeb414ed9957aa3a5e43a869
Best match: python-magic 0.4.13
Processing python-magic-0.4.13.tar.gz
Writing /tmp/easy_install-8wsjmu97/python-magic-0.4.13/setup.cfg
Running python-magic-0.4.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-8wsjmu97/python-magic-0.4.13/egg-dist-tmp-ogmrf50a
zip_safe flag not set; analyzing archive contents...
Moving python_magic-0.4.13-py3.6.egg to /home/ele/develop/packaging/devel:languages:python/python-pymisp/pymisp-2.4.81.2/.eggs

Installed /home/ele/develop/packaging/devel:languages:python/python-pymisp/pymisp-2.4.81.2/.eggs/python_magic-0.4.13-py3.6.egg
running egg_info
writing pymisp.egg-info/PKG-INFO
writing dependency_links to pymisp.egg-info/dependency_links.txt
writing requirements to pymisp.egg-info/requires.txt
writing top-level names to pymisp.egg-info/top_level.txt
reading manifest file 'pymisp.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pymisp.egg-info/SOURCES.txt'
running build_ext
test_addAttributes (tests.test_offline.TestOffline) ... The version of PyMISP recommended by the MISP instance (2.4.62) is older than the one you're using now (2.4.81.2). Please upgrade the MISP instance or use an older PyMISP version.
ok
test_auth_error (tests.test_offline.TestOffline) ... The version of PyMISP recommended by the MISP instance (2.4.62) is older than the one you're using now (2.4.81.2). Please upgrade the MISP instance or use an older PyMISP version.
ok
test_deleteAttribute (tests.test_offline.TestOffline) ... The version of PyMISP recommended by the MISP instance (2.4.62) is older than the one you're using now (2.4.81.2). Please upgrade the MISP instance or use an older PyMISP version.
ERROR
test_deleteEvent (tests.test_offline.TestOffline) ... The version of PyMISP recommended by the MISP instance (2.4.62) is older than the one you're using now (2.4.81.2). Please upgrade the MISP instance or use an older PyMISP version.
ok
test_eventObject (tests.test_offline.TestOffline) ... The version of PyMISP recommended by the MISP instance (2.4.62) is older than the one you're using now (2.4.81.2). Please upgrade the MISP instance or use an older PyMISP version.
ok
test_getEvent (tests.test_offline.TestOffline) ... The version of PyMISP recommended by the MISP instance (2.4.62) is older than the one you're using now (2.4.81.2). Please upgrade the MISP instance or use an older PyMISP version.
ok
test_getSharingGroups (tests.test_offline.TestOffline) ... The version of PyMISP recommended by the MISP instance (2.4.62) is older than the one you're using now (2.4.81.2). Please upgrade the MISP instance or use an older PyMISP version.
ok
test_getVersions (tests.test_offline.TestOffline) ... The version of PyMISP recommended by the MISP instance (2.4.62) is older than the one you're using now (2.4.81.2). Please upgrade the MISP instance or use an older PyMISP version.
ok
test_newEvent (tests.test_offline.TestOffline) ... The version of PyMISP recommended by the MISP instance (2.4.62) is older than the one you're using now (2.4.81.2). Please upgrade the MISP instance or use an older PyMISP version.
ok
test_objects (tests.test_offline.TestOffline) ... /home/ele/develop/packaging/devel:languages:python/python-pymisp/pymisp-2.4.81.2/pymisp/tools/fileobject.py:30: UserWarning: Please install pydeep: pip install git+https://github.com/kbandla/pydeep.git
  warnings.warn("Please install pydeep: pip install git+https://github.com/kbandla/pydeep.git")
/home/ele/develop/packaging/devel:languages:python/python-pymisp/pymisp-2.4.81.2/pymisp/tools/fileobject.py:32: UserWarning: Please install python-magic: pip install python-magic.
  warnings.warn("Please install python-magic: pip install python-magic.")
ok
test_searchIndexByTagId (tests.test_offline.TestOffline) ... The version of PyMISP recommended by the MISP instance (2.4.62) is older than the one you're using now (2.4.81.2). Please upgrade the MISP instance or use an older PyMISP version.
ok
test_searchIndexByTagName (tests.test_offline.TestOffline) ... The version of PyMISP recommended by the MISP instance (2.4.62) is older than the one you're using now (2.4.81.2). Please upgrade the MISP instance or use an older PyMISP version.
ok
test_updateEvent (tests.test_offline.TestOffline) ... The version of PyMISP recommended by the MISP instance (2.4.62) is older than the one you're using now (2.4.81.2). Please upgrade the MISP instance or use an older PyMISP version.
ok

======================================================================
ERROR: test_deleteAttribute (tests.test_offline.TestOffline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ele/develop/packaging/devel:languages:python/python-pymisp/pymisp-2.4.81.2/.eggs/requests_mock-1.3.0-py3.6.egg/requests_mock/mocker.py", line 219, in inner
    return func(*args, **kwargs)
  File "/home/ele/develop/packaging/devel:languages:python/python-pymisp/pymisp-2.4.81.2/tests/test_offline.py", line 82, in test_deleteAttribute
    d = pymisp.delete_attribute(2)
  File "/home/ele/develop/packaging/devel:languages:python/python-pymisp/pymisp-2.4.81.2/pymisp/api.py", line 370, in delete_attribute
    response = session.get(url)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ele/develop/packaging/devel:languages:python/python-pymisp/pymisp-2.4.81.2/.eggs/requests_mock-1.3.0-py3.6.egg/requests_mock/mocker.py", line 101, in _fake_send
    return _original_send(session, request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/home/ele/develop/packaging/devel:languages:python/python-pymisp/pymisp-2.4.81.2/.eggs/requests_mock-1.3.0-py3.6.egg/requests_mock/adapter.py", line 233, in send
    raise exceptions.NoMockAddress(request)
requests_mock.exceptions.NoMockAddress: No mock address: GET http://misp.local/attributes/delete/2

----------------------------------------------------------------------
Ran 13 tests in 0.195s

FAILED (errors=1)
Test failed: <unittest.runner.TextTestResult run=13 errors=1 failures=0>
error: Test failed: <unittest.runner.TextTestResult run=13 errors=1 failures=0>

The mocks in initURI do not have this URI.

Correlate toggle dialog doesn't display if clicked after attribute add

• Ubuntu Linux 14.04.5 LTS
• PHP 5.5.9-1ubuntu4.21
• MISP 2.4.69

Having some trouble with a behavior that I think shows up like this:

1. Add new attribute to an event
2. Click to toggle the Correlate checkbox on the new attribute
3. The screen dims as the confirmation dialog renders, but the dialog (modal?) is not rendered. Hit escape to dismiss the invisible dialog. Any number of attempts to display dialog upon checking box fail.
4. To fix, reload the event URL and allow page to re-render
5. Click the Correlate checkbox and the confirmation dialog renders and functions correctly

Calls to PyMISP.update with wrong event argument

I think 619538c introduced a bug in:

• PyMISP.publish
• PyMISP.change_threat_level
• PyMISP.change_sharing_group
• PyMISP._send_attributes

Example:

def change_threat_level(self, event, threat_level_id):
        e = self._make_mispevent(event)
        e.threat_level_id = threat_level_id
        return self.update(event)

self.update should be called with e as argument, not event.

Comment and Distribution no effect in upload_sample / upload_samplelist

The upload_sample and upload_samplelist routines bot accect a comment and distribution argument. However, they do not have any effect. The comment is never set, and the distribution of the sample is set to Organization.

You can reproduce the issue with upload.py in the example folder:

python3 upload.py -u upload.py -e 200 --distrib 3 --comment "Comment"

Tested with commit 14bc9e4, Thu Oct 26 12:05:51 2017 -0400.

Please keep the changelog up to date

The last two entries are

• %%version%% (unreleased)
• v2.4.71 (2017-04-11)

while the current release is 2.4.79

Sample file altered when using upload_sample

The sample with correct hash '5dbf819539ae860f0d52797ebf0a3a5f' is uploaded via 'Add Attachment' whereas the other one is via PyMISP upload_sample.

Code used:
r = m.upload_sample(sample_fname, '/some/path/' + sample_fname, None, None, True, None, None, row[1], None, severity)

Code works since a new event is created with attributes related to the uploaded file.

MISP 2.4.67
pymisp-2.4.67-py3.4

Setting to_ids=False has no effect

Maybe this is a bug in PyMISP: When adding an attribute to an object and setting this attribute the flag to_ids to False then the MISP does not set it to False when it is by default True.

Add support for attachTagToObject

Doc: MISP/MISP@76c0cb4

How to know if a event has been deleted or updated?

I have fetched 1 month data using PyMISP and stored locally. Now I want to fetch incremental data only eg. 1 day. This can be easily done using download_last('time_interval') functionality given by the API. I want to know is there anyway to find out whether the events in last month data has been updated or deleted so that I can change my locally stored data.

Thanking in advance. Please let me know if the question needs further explanation.

Issues with pypi six

After cloning the repository I couldn't pip install it. Here is the command (pip3 install -I . gave me ssl error that's why i put the --truste-host and index-url).

root@host[/var/www/MISP/PyMISP/examples]: pip3 install -I . --trusted-host pypi.python.org --index-url=http://pypi.python.org/simple/six/
Processing /var/www/MISP/PyMISP
Collecting six (from pymisp==2.4.81.2)
Could not find a version that satisfies the requirement six (from pymisp==2.4.81.2) (from versions: )
No matching distribution found for six (from pymisp==2.4.81.2)

Composites are not taken into account

I make some update to openioc tool in PyMISP
https://github.com/truckydev/PyMISP/blob/master/pymisp/tools/openioc.py

My code detect composites supported by misp.
It may need some refactoring :)

Here is my problem :
When I try with this IOC :
https://github.com/fireeye/iocs/blob/master/BlogPosts/9cee306d-5441-4cd3-932d-f3119752634c.ioc

<Indicator id="4b8e8c05-a8f9-4aff-9b7b-bd7301e1cfd7" operator="AND">
        <IndicatorItem id="317dc9d2-bd92-4fb1-87b1-cb5a158a4828" condition="is">
          <Context document="FileItem" search="FileItem/Md5sum" type="mir"/>
          <Content type="md5">1F338BDD92F08803A2AC7022A34D98FD</Content>
        </IndicatorItem>
        <IndicatorItem id="ec96b850-238a-4b1a-a40e-717b88522191" condition="is">
          <Context document="FileItem" search="FileItem/FileName" type="mir"/>
          <Content type="string">install.exe</Content>
        </IndicatorItem>
      </Indicator>

when I print ( with main in python code) I have the correct type

{'type': 'filename|md5', 'comment': '', 'to_ids': True, 'distribution': '5', 'category': 'External analysis', 'value': 'install.exe|1F338BDD92F08803A2AC7022A34D98FD', 'disable_correlation': False}

but with Openioc module import, this type is not recognized and add element line by line

add_tag(event...) is broken!

If you pass an event to add_tag, it will get an index error because:

2bcc416

to_post = {'request': {'Event': {'id': event['Event']['id'], 'tag': tag}}}

was changed to:

 +            to_post = {'request': {'Event': {'id': event['id'], 'tag': tag}}} 

So there is a missing ['Event']

Will create a PR for it. For remove_tag it is okay.

Upload attachment to event (not sample)

Hi,

I am trying to upload a file into a MISP event, for obvious reasons I want store external PDF reports directly in MISP so I dont have to rely on links being available when and where I need them.

For simplicity's sake I can reproduce the problem with the examples/up.py code.
(my own code is built the same way and I get the same error).

When running the up.py -e 29 -i pdf.pdf I get the following error.

UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 600: character maps to

Process finished with exit code 1

(so probably related to encoding)

I assume that it should be possible to upload files directly into MISP that are not samples and therefore should not be hashed, password protected etc. (upload.py works fine but I assume that it is more targeted at samples)

I am unsure if up.py is the correct example to use, but I get the same error if I use misp.add_attachment function in pymisp.

events/edit API readds deleted attributes already returned by a search

Steps to reproduce:

• fetch an event (with id: event_id) with my_event = misp.search()
• delete some attributes from event_id
• add an attribute with misp.add_attribute(my_event, )
  => the deleted attributes are recreated as they are still included in my_event

workaround:
set my_event['Event']['Attribute'] = [ ] before adding any new attributes.

possible fixes:

• create a different API which supports deltas which can be uploaded in one go
• on adding an attribute do not resend the old attributes

adding an attribtue to an event changes timestamps of all attributes, adding shadow attributes crahses

Whenever an attribute is added to an event, PyMisp obviously exports the full event data and uploads the full data again, which causes large queries for only small changed. More importantly, it seems that all timestamps of all attributes in the event are changed to now, which basically destroys the whole timeline history. I'm not sure if this is a bug of PyMisp itself, or a problem of the REST API, where I don't see a possibility to add an attribute (only shadow attributes) to an event.

also, adding shadow attributes via PyMisp (proposal=True) always crashes, it seems teh attributes object is not serializable.

Misp_verifycert is not implemented into examples

Hi,

During test, I saw in examples that "misp_verifycert" (one of keys.py value) isn't used when calling PyMisp.

Could you update Examples files ?

get_sharing_groups() only returns the first group

The API function get_sharing_groups() only returns the first entry of the list of sharing groups:
return self._check_response(response)['response'][0]

I guess it should be changed to:
return self._check_response(response)['response']

add_file_object.py against some field is not creating some attribute in misp object

First file:
python3 add_file_object.py -e 6683-p /user/analyst/binaries/7baa0260b1e98a21762818aeea6b73c32a3b68787bedbc08aac276170cc55a50
misp file object created with no errors

Second file:
python3 add_file_object.py -e 6683 -p /user/analyst/binaries/fdab41ae51989ca6cf31071b8ec3b5ae9106466548b57b7faee38fcf52f32ead
misp file ojbect created with missing attributes and this error:

Traceback (most recent call last):
File "add_file_object.py", line 41, in
response = pymisp.add_object(args.event, template_id, fo)
File "/usr/local/lib/python3.4/dist-packages/pymisp/api.py", line 1628, in add_object
return self._check_response(response)
File "/usr/local/lib/python3.4/dist-packages/pymisp/api.py", line 210, in _check_response
response.raise_for_status()
File "/usr/local/lib/python3.4/dist-packages/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://1.2.3.4/objects/add/6683/9

It seems that no issue with:
exe, pdf, dat.

Issue found with:
Jar files with mimetype: Java Jar file data (zip)
Zip files with mimetype: Zip archive data, at least v1.0 to extract

Update search API

MISP/MISP@8c63e6f

sighting_per_uuid returning error with wrong URL

Hi

Im adding some functionality to an application im using, but when i pass the uuid to add a sighting i get this

misp.sighting_per_uuid('290cfb09-702d-4a1a-92e6-b2f10d857695')
{u'url': u'/sighting/add/290cfb09-702d-4a1a-92e6-b2f10d857695', u'message': u'Could not add Sighting', u'errors': [u'Could not add Sighting'], u'name': u'Could not add Sighting'}

I had a look in the pymisp api and it should be posting to /sightings/add/[uuid] but the error im getting seems to be missing th "S" from sightings and im wondering if this may be a bug ?

im using
pymisp 2.4.56
ubuntu 16.04
misp 2.4.75

Cheers.

NameError: name 'FuturesSession' is not defined

if requests-futures is not installed https://github.com/MISP/PyMISP/blob/master/pymisp/api.py#L780 will produce a NameError error.

Using try: FuturesSession or something else would be great to avoid this.

get_all_tags() only returns 50 results (first page)

When invoking PyMISP get_all_tags, only the first 50 results are returned (first page).

The REST api would allow you to get subsequent pages with the tags/index/page:n URL, but there is no way to specify it in PyMISP.

Is there any workaround? Any way to get the details for a specific tag?

Disable correlation for attribute

Hi,
we have developed a script to create events and add attributes to them from an external source.
Now, we'd like to disable correlation on some attributes inside the script, but we cannot find anything in the API to do it.
Is this possible?
Thanks!!!
M.

bug in type check condition of API search call

def __prepare_rest_search(self, values, not_values): to_return = '' if values is not None: if not isinstance(values, list): to_return += values else: to_return += '&&'.join(values) if not_values is not None: if len(to_return) > 0: to_return += '&&!' else: to_return += '!' if not isinstance(values, list): #should be if not isinstance(not_values, list) to_return += not_values else: to_return += '&&!'.join(not_values) return to_return

Expected

when filtering only by a list of not_value the query should succeed

Behaviour

query fails with type error as in the code snippet above in the not_value checks isinstance check the "values" type instead of the "not_values" type hence the call will fail with a type error if the type of values is different than that of the not_values or if the value is not specified and not_values is an array

Recreate

use search("events", not_tags=["a", "b"])
will fail with "TypeError: cannot concatenate 'str' and 'list' objects"