csirtgadgets / cifsdk-v2 Goto Github PK

View Code? Open in Web Editor NEW

13.0 13.0 5.0 259 KB

python SDK for CIFv2

Home Page: http://csirtgadgets.org/collective-intelligence-framework/

License: GNU Lesser General Public License v3.0

Python 99.32% Ruby 0.68%

cif python

cifsdk-v2's People

Contributors

Stargazers

Watchers

Forkers

aeppert ajt fl0x2208 nibor62 sfinlon

cifsdk-v2's Issues

Force passing dict or list of dicts

https://github.com/csirtgadgets/py-cifsdk/blob/master/cifsdk/client.py#L136

if instance data, dict
else
elif instance data[0] not a dict
else:
raise error: cannot pass string

better errors when config is missing

raceback (most recent call last):
  File "/usr/local/bin/cg-cif", line 9, in <module>
    load_entry_point('py-cgcif==0.0.0a2-5-g85d4827', 'console_scripts', 'cg-cif')()
  File "/usr/local/lib/python2.7/dist-packages/cgcif/__init__.py", line 112, in main
    if not options['token']:
KeyError: 'token'

less verbose error message when missing/incorrect config

cif -q example.com -C /home/giovino/.ciff.yml
Traceback (most recent call last):
  File "/home/giovino/code/py-cifsdk/venv/bin/cif", line 9, in <module>
    load_entry_point('py-cifsdk==0.0.3-20-ge4992ad', 'console_scripts', 'cif')()
  File "/home/giovino/code/py-cifsdk/py-cifsdk/cifsdk/client.py", line 245, in main
    o = read_config(args)
  File "/home/giovino/code/py-cifsdk/py-cifsdk/cifsdk/utils.py", line 22, in read_config
    raise Exception("Unable to read {} config file".format(args.config))
Exception: Unable to read /home/giovino/.ciff.yml config file

Not raise an Exception?

enable feed aggregation with --feed command

--today argument missing

perl client

--today                     auto-sets reporttime to today, 00:00:00Z (UTC)

--last-hour argument missing

perl

    --last-hour                 auto-sets reporttime to the beginning of the previous full hour
                                and reporttime-end to end of previous full hour

csv output truncation bug

the cifv2 python client truncates the url in csv output when it should only be truncated in table output.

truncation example:

$ cif --feed --otype url --confidence 85 --tags botnet --fields tlp,group,lasttime,observable,confidence,description,tags,provider --limit 1 -f csv
limited,everyone,2016-04-06T15:04:27Z,http://cp53072.cloudhosting.lv..,85,dropzone,"zeus,botnet",zeustracker.abuse.ch

no truncation example:

$ cif --feed --otype url --confidence 85 --tags botnet --fields tlp,group,lasttime,observable,confidence,description,tags,provider --limit 1 -f json | python -m json.tool
[
    {
        "altid": "https://zeustracker.abuse.ch/monitor.php?search=http://cp53072.cloudhosting.lv/bord/gate.php",
        "confidence": 85,
        "description": "dropzone",
        "firsttime": "2016-04-06T15:04:27Z",
        "group": [
            "everyone"
        ],
        "id": "815058255cc4e974d764762c0edafef1a54be3cb35f7e85560ce161245854cae",
        "lang": "EN",
        "lasttime": "2016-04-06T15:04:27Z",
        "observable": "http://cp53072.cloudhosting.lv/bord/gate.php",
        "otype": "url",
        "provider": "zeustracker.abuse.ch",
        "reporttime": "2016-04-06T15:04:41Z",
        "tags": [
            "zeus",
            "botnet"
        ],
        "tlp": "limited"
    }
]

snort output formatter

passing --feed, default confidence of 65

unclear how to use -v

cif -q example.com -l 3 -v 1
usage: cif [-h] [-v] [-d] [-V] [--no-verify-ssl] [--remote REMOTE]
           [--token TOKEN] [--timeout TIMEOUT] [-C CONFIG] [--sort SORT]
           [-f FORMAT] [-p] [-s SUBMIT] [-l LIMIT] [-n] [-q QUERY]
           [--firsttime FIRSTTIME] [--lasttime LASTTIME]
           [--reporttime REPORTTIME] [--reporttimeend REPORTTIMEEND]
           [--tags TAGS] [--description DESCRIPTION] [--otype OTYPE] [--cc CC]
           [-c CONFIDENCE] [--rdata RDATA] [--provider PROVIDER] [--asn ASN]
           [--proxy PROXY] [--feed] [--whitelist-limit WHITELIST_LIMIT]
           [--last-day] [--days DAYS] [--aggregate AGGREGATE]
cif: error: unrecognized arguments: 1

Missing '--last-day' cli arguments

The cli is missing the '--last-day' argument seen in the perl client.

try/except for decompression fails on older instances when larger amounts of data

2016-01-25 09:04:21,243 - DEBUG - cifsdk.client[92] - status code: 200
2016-01-25 09:04:21,243 - INFO - cifsdk.client[99] - processing 14 megs
2016-01-25 09:04:21,243 - INFO - cifsdk.client[102] - trying to decompress...

AttributeError: 'str' object has no attribute 'get'

ubuntu@data:~$ cif --provider example.org
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
2015-08-18 15:00:40,673 - ERROR - cifsdk.client::MainThread - request failed: 500
2015-08-18 15:00:40,673 - ERROR - cifsdk.client::MainThread - 'str' object has no attribute 'get'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cifsdk/client.py", line 270, in main
    print(f(ret))
  File "/usr/local/lib/python2.7/dist-packages/cifsdk/format/table.py", line 16, in __repr__
    y = obs.get(c) or ''
AttributeError: 'str' object has no attribute 'get'

command line arguments

The following command line arguments are not the same from the perl client.

-c, -f are not short for --confidence and --format which makes it somewhat confusing going between those two clients.

I did not analyse other potential differences, this issue is to capture the need to do that analysis.

--feed should raise error if whitelist feed is empty

at-least for fqdn and ipv4

--fields argument missing

perl

--fields                    specify output fields [default: tlp,group,reporttime,observable,cc,asn,confidence,tags,description,rdata,provider,altid_tlp,altid]

TypeError: object of type 'float' has no len()

$ cif -q example.com -f csv
Traceback (most recent call last):
  File "/usr/local/bin/cif", line 9, in <module>
    load_entry_point('py-cifsdk==2.0.0a2', 'console_scripts', 'cif')()
  File "/usr/local/lib/python2.7/dist-packages/cifsdk/client.py", line 415, in main
    print(ret)
  File "/usr/local/lib/python2.7/dist-packages/cifsdk/format/cifcsv.py", line 27, in __repr__
    y = (y[:self.max_field_size] + '..') if len(y) > self.max_field_size else y
TypeError: object of type 'float' has no len()

ipv4 feed error

when running the whitelist_testing.sh script

#verify ipv4
tmp=`$CIF_CMD --token $EVERYONE_TOKEN --feed --otype ipv4 -c 85 -f csv | grep '$IPV4'`
if [ $? != 1 ]; then echo "ERROR: $IPV4 does not exist for everyone in ipv4 feed"; fi

got this error:

Traceback (most recent call last):
  File "/usr/local/bin/cif", line 9, in <module>
    load_entry_point('cif-sdk-py==2.0.0a2-12-gda15b3d', 'console_scripts', 'cif')()
  File "/usr/local/lib/python2.7/dist-packages/cifsdk/client.py", line 407, in main
    ret = f().process(ret, wl)
  File "/usr/local/lib/python2.7/dist-packages/cifsdk/feed/ipv4.py", line 49, in process
    if str(y['observable']) not in wl:
ValueError: Error parsing prefix.

doc fixes

api needs actual pydoc strings...

I have a question about the Python SDK search example given on https://github.com/csirtgadgets/py-cifsdk

First, I had to change noverifyssl=1 to no_verify_ssl=1 - this looks like a typo

Secondly,  when I use ret = cli.search(query='example.com') then the script returns the following :-
Traceback (most recent call last):
File "rch-cif-test.py", line 6, in <module>
ret = cli.search(query='example.com')
TypeError: search() got an unexpected keyword argument 'query'

Can anyone help clarify how to correctly use the Python cli.search method ?

csv formatter

--sort does not mirror the perl client

perl

    --sortby                    sort output, default: lasttime
    --sortby-direction          sortby direction, default: asc

cif -q broken

cif -q example.com
2015-11-18 06:16:30,173 - WARNING - cifsdk.client[361] - operation not supported

Needs to be updated for argument change
https://github.com/csirtgadgets/py-cifsdk/blob/master/cifsdk/client.py#L261

--feed errors

cif --otype fqdn -c 85 --feed -f csv
2015-11-18 13:55:31,037 - ERROR - cifsdk.client[368] - unhashable type: 'dict'

cif --otype url -c 85 --feed -f csv
2015-11-18 13:56:55,667 - ERROR - cifsdk.client[368] - global name 'tag_contains_whitelist' is not defined

cif --otype ipv4 -c 85 --feed -f csv
2015-11-18 13:58:10,338 - ERROR - cifsdk.client[368] - 'ascii' codec can't encode characters in position 5-6: ordinal not in range(128)

cif --otype email -c 85 --feed -f csv
2015-11-18 13:58:57,740 - ERROR - cifsdk.client[368] - 'NoneType' object is not callable

Error: No handlers could be found for logger "cifsdk.client"

using the wrong token surfaces this error:

code:

import time                                                                                                                                                                                                            
from cifsdk.client import Client
import json
from pprint import pprint

cli = Client(token='wrong',
             remote='https://192.168.1.210',
             verify_ssl=False)


ret = cli.search('example.com')
pprint(ret)

error:

$ python wf-submit.py 
No handlers could be found for logger "cifsdk.client"

--id <id> search missing from cli

in the perl sdk we have the following:

--id STRING                 specify an id to retrieve

do not write output if no results

in the perl client we do not write an output table when there are zero results. in the python client we do. mirror the perl client?

cif -q example112.com -l 3
-----+----------+------------+-------------+------+-------+----------+
| tlp | reporttime | observable | otype | cc | asn | asn_desc | confidence | description | tags | rdata | provider |
+-----+------------+------------+-------+----+-----+----------+------------+-------------+------+-------+----------+
+-----+------------+------------+-------+----+-----+----------+------------+-------------+------+-------+----------+

permissions error on install

the install directions to not say to sudo pip install...

which results in this error:

error: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pytricia.so'

----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_giovino/pytricia/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-X4tgUX-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_giovino/pytricia

Need to update the install instructions with:

sudo pip install git+https://github.com/csirtgadgets/py-cifsdk

bind output

catch return output

python sumbit.py 
['{"observable":"example4.com","tlp":"amber","confidence":"85","tags":"malware","provider":"me.com","group":"everyone"}', '{"observable":"example4.com","tlp":"amber","confidence":"85","tags":"malware","provider":"me.com","group":"everyone"}']
No handlers could be found for logger "cifsdk.client"
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "sumbit.py", line 23, in worker
    ret = cli.submit(data)
  File "/home/giovino/py-submit/venv/local/lib/python2.7/site-packages/cifsdk/client.py", line 130, in submit
    self.logger.error(json.loads(body.text).get('message'))
ValueError: Expected object or value

test 2.7.6 for basestring

https://github.com/csirtgadgets/py-cifsdk/blob/master/cifsdk/client.py#L113

ipv6 feed error

when running the whiltelist_testing.sh script got the following error:

#verify ipv6
tmp=`$CIF_CMD --token $EVERYONE_TOKEN --feed --otype ipv6 -c 85 -f csv | grep '$IPV6'`
if [ $? != 1 ]; then echo "ERROR: $IPV6 does not exist for everyone in ipv6 feed"; fi

Traceback (most recent call last):
  File "/usr/local/bin/cif", line 9, in <module>
    load_entry_point('cif-sdk-py==2.0.0a2-12-gda15b3d', 'console_scripts', 'cif')()
  File "/usr/local/lib/python2.7/dist-packages/cifsdk/client.py", line 407, in main
    ret = f().process(ret, wl)
  File "/usr/local/lib/python2.7/dist-packages/cifsdk/feed/ipv6.py", line 24, in process
    raise RuntimeError('https://github.com/jsommers/pytricia/issues/6')
RuntimeError: https://github.com/jsommers/pytricia/issues/6

snort output

--whitelist-confidence argument missing

perl

--whitelist-confidence=INT  by confidence (greater or equal to) (default 25)

License mismatch

README.md says the content is LGPL licensed, but LICENSE says Apache

--submit is not working

probably need to store true here:
https://github.com/csirtgadgets/py-cifsdk/blob/master/cifsdk/client.py#L210

probably need to read in "stdin" here:
https://github.com/csirtgadgets/py-cifsdk/blob/master/cifsdk/client.py#L352

and submit "stdin"...

The question is, stdin is a string not a dist or list of dicts, what's the best want to convert that string? In the perl sdk we expect json and json'ify it.

Missing '--description' cli argument

The cli is missing the '--description' argument seen in the perl client.

no email tests exist for for text/plain

Installation instructions miss some requirements for client to run

Following the install instructions results in an ImportError when testing the client:

ImportError: No module named prettytable

Easily fixed with:
sudo pip install PrettyTable

This was on an Ubuntu 12 LTS box.

flowspec output plugin

try/catch when no data

wes@cif:~$ cif -d --tags phishing --provider apwg --limit 5 --otype url
2015-04-29 11:39:04,127 - DEBUG - cifsdk.client::MainThread - uri: https://localhost/observables
2015-04-29 11:39:04,127 - DEBUG - cifsdk.client::MainThread - params: {"nolog":null,"otype":"url","provider":"apwg","limit":"5","tags":"phishing"}
2015-04-29 11:39:04,127 - INFO - cifsdk.client::MainThread - searching...
2015-04-29 11:39:04,141 - INFO - requests.packages.urllib3.connectionpool::MainThread - Starting new HTTPS connection (1): localhost
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
2015-04-29 11:39:04,167 - ERROR - cifsdk.client::MainThread - connection error
2015-04-29 11:39:04,168 - ERROR - cifsdk.client::MainThread - 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cifsdk/client.py", line 258, in main
    print(Table(ret))
  File "/usr/local/lib/python2.7/dist-packages/cifsdk/format/table.py", line 24, in __repr__
    y = obs.get(c) or ''
AttributeError: 'NoneType' object has no attribute 'get'

long form of -q not the same as the perl client

perl cli

    -q, --query=STRING          specify a search

python cli

  -q SEARCH, --search SEARCH
                        search for observable

we should standardize.. --query or --search

filter by asn

json formatter

--ttl argument missing

perl

    --ttl=INT                   specify number of pings to send, default: 4
                                (0 infinite, halt with SIGINT or CTRL+C)

--filename argument missing

perl

--filename                  specify output filename [default: STDOUT]

bro formatter

/usr/local/bin/cif overwrites perl version

When installing py-cifsdk on a CIF server, py-cifsdk overwrites the perl version of /usr/local/bin/cif.

Is this the expected behavior? Seems not ideal to me although I fully realize then you you have to deal with different names for the cif bin.

-g

--group argument missing

perl client

    --group=STRING              by groups (everyone,group1,group2, ...)

email.py does not handle non-multi-part messages

this does not handle non-mulitpart messages

body.append(parse_message_part(msg))

https://github.com/csirtgadgets/py-cifsdk/blob/master/cifsdk/email.py#L136

it has to be something like:

        if msg.get_default_type() == "text/plain":
            print("get_payload-1:", msg.get_payload(decode=True))
        elif msg.get_default_type() == "text/html":
            print("get_payload-2:", msg.get_payload(decode=True))
        else:
            # not sure a non mulitpart message has anything be text/plain and text/html
            print("WARNING: unhandled default.type", msg.get_default_type())

--application argument missing

This exists in the perl client, not in the python client.

perl

    --application=STRING        filter based on the application field

csirtgadgets / cifsdk-v2 Goto Github PK

cifsdk-v2's People

Contributors

Stargazers

Watchers

Forkers

cifsdk-v2's Issues

Recommend Projects

Recommend Topics

Recommend Org