csirtgadgets / cifsdk-v2 Goto Github PK
View Code? Open in Web Editor NEWpython SDK for CIFv2
Home Page: http://csirtgadgets.org/collective-intelligence-framework/
License: GNU Lesser General Public License v3.0
python SDK for CIFv2
Home Page: http://csirtgadgets.org/collective-intelligence-framework/
License: GNU Lesser General Public License v3.0
https://github.com/csirtgadgets/py-cifsdk/blob/master/cifsdk/client.py#L136
if instance data, dict
else
elif instance data[0] not a dict
else:
raise error: cannot pass string
raceback (most recent call last):
File "/usr/local/bin/cg-cif", line 9, in <module>
load_entry_point('py-cgcif==0.0.0a2-5-g85d4827', 'console_scripts', 'cg-cif')()
File "/usr/local/lib/python2.7/dist-packages/cgcif/__init__.py", line 112, in main
if not options['token']:
KeyError: 'token'
cif -q example.com -C /home/giovino/.ciff.yml
Traceback (most recent call last):
File "/home/giovino/code/py-cifsdk/venv/bin/cif", line 9, in <module>
load_entry_point('py-cifsdk==0.0.3-20-ge4992ad', 'console_scripts', 'cif')()
File "/home/giovino/code/py-cifsdk/py-cifsdk/cifsdk/client.py", line 245, in main
o = read_config(args)
File "/home/giovino/code/py-cifsdk/py-cifsdk/cifsdk/utils.py", line 22, in read_config
raise Exception("Unable to read {} config file".format(args.config))
Exception: Unable to read /home/giovino/.ciff.yml config file
Not raise an Exception?
perl client
--today auto-sets reporttime to today, 00:00:00Z (UTC)
perl
--last-hour auto-sets reporttime to the beginning of the previous full hour
and reporttime-end to end of previous full hour
the cifv2 python client truncates the url in csv output when it should only be truncated in table output.
truncation example:
$ cif --feed --otype url --confidence 85 --tags botnet --fields tlp,group,lasttime,observable,confidence,description,tags,provider --limit 1 -f csv
limited,everyone,2016-04-06T15:04:27Z,http://cp53072.cloudhosting.lv..,85,dropzone,"zeus,botnet",zeustracker.abuse.ch
no truncation example:
$ cif --feed --otype url --confidence 85 --tags botnet --fields tlp,group,lasttime,observable,confidence,description,tags,provider --limit 1 -f json | python -m json.tool
[
{
"altid": "https://zeustracker.abuse.ch/monitor.php?search=http://cp53072.cloudhosting.lv/bord/gate.php",
"confidence": 85,
"description": "dropzone",
"firsttime": "2016-04-06T15:04:27Z",
"group": [
"everyone"
],
"id": "815058255cc4e974d764762c0edafef1a54be3cb35f7e85560ce161245854cae",
"lang": "EN",
"lasttime": "2016-04-06T15:04:27Z",
"observable": "http://cp53072.cloudhosting.lv/bord/gate.php",
"otype": "url",
"provider": "zeustracker.abuse.ch",
"reporttime": "2016-04-06T15:04:41Z",
"tags": [
"zeus",
"botnet"
],
"tlp": "limited"
}
]
cif -q example.com -l 3 -v 1
usage: cif [-h] [-v] [-d] [-V] [--no-verify-ssl] [--remote REMOTE]
[--token TOKEN] [--timeout TIMEOUT] [-C CONFIG] [--sort SORT]
[-f FORMAT] [-p] [-s SUBMIT] [-l LIMIT] [-n] [-q QUERY]
[--firsttime FIRSTTIME] [--lasttime LASTTIME]
[--reporttime REPORTTIME] [--reporttimeend REPORTTIMEEND]
[--tags TAGS] [--description DESCRIPTION] [--otype OTYPE] [--cc CC]
[-c CONFIDENCE] [--rdata RDATA] [--provider PROVIDER] [--asn ASN]
[--proxy PROXY] [--feed] [--whitelist-limit WHITELIST_LIMIT]
[--last-day] [--days DAYS] [--aggregate AGGREGATE]
cif: error: unrecognized arguments: 1
The cli is missing the '--last-day' argument seen in the perl client.
2016-01-25 09:04:21,243 - DEBUG - cifsdk.client[92] - status code: 200
2016-01-25 09:04:21,243 - INFO - cifsdk.client[99] - processing 14 megs
2016-01-25 09:04:21,243 - INFO - cifsdk.client[102] - trying to decompress...
ubuntu@data:~$ cif --provider example.org
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
2015-08-18 15:00:40,673 - ERROR - cifsdk.client::MainThread - request failed: 500
2015-08-18 15:00:40,673 - ERROR - cifsdk.client::MainThread - 'str' object has no attribute 'get'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cifsdk/client.py", line 270, in main
print(f(ret))
File "/usr/local/lib/python2.7/dist-packages/cifsdk/format/table.py", line 16, in __repr__
y = obs.get(c) or ''
AttributeError: 'str' object has no attribute 'get'
The following command line arguments are not the same from the perl client.
-c, -f are not short for --confidence and --format which makes it somewhat confusing going between those two clients.
I did not analyse other potential differences, this issue is to capture the need to do that analysis.
at-least for fqdn and ipv4
perl
--fields specify output fields [default: tlp,group,reporttime,observable,cc,asn,confidence,tags,description,rdata,provider,altid_tlp,altid]
$ cif -q example.com -f csv
Traceback (most recent call last):
File "/usr/local/bin/cif", line 9, in <module>
load_entry_point('py-cifsdk==2.0.0a2', 'console_scripts', 'cif')()
File "/usr/local/lib/python2.7/dist-packages/cifsdk/client.py", line 415, in main
print(ret)
File "/usr/local/lib/python2.7/dist-packages/cifsdk/format/cifcsv.py", line 27, in __repr__
y = (y[:self.max_field_size] + '..') if len(y) > self.max_field_size else y
TypeError: object of type 'float' has no len()
when running the whitelist_testing.sh script
#verify ipv4
tmp=`$CIF_CMD --token $EVERYONE_TOKEN --feed --otype ipv4 -c 85 -f csv | grep '$IPV4'`
if [ $? != 1 ]; then echo "ERROR: $IPV4 does not exist for everyone in ipv4 feed"; fi
got this error:
Traceback (most recent call last):
File "/usr/local/bin/cif", line 9, in <module>
load_entry_point('cif-sdk-py==2.0.0a2-12-gda15b3d', 'console_scripts', 'cif')()
File "/usr/local/lib/python2.7/dist-packages/cifsdk/client.py", line 407, in main
ret = f().process(ret, wl)
File "/usr/local/lib/python2.7/dist-packages/cifsdk/feed/ipv4.py", line 49, in process
if str(y['observable']) not in wl:
ValueError: Error parsing prefix.
api needs actual pydoc strings...
I have a question about the Python SDK search example given on https://github.com/csirtgadgets/py-cifsdk
First, I had to change noverifyssl=1 to no_verify_ssl=1 - this looks like a typo
Secondly, when I use ret = cli.search(query='example.com') then the script returns the following :-
Traceback (most recent call last):
File "rch-cif-test.py", line 6, in <module>
ret = cli.search(query='example.com')
TypeError: search() got an unexpected keyword argument 'query'
Can anyone help clarify how to correctly use the Python cli.search method ?
perl
--sortby sort output, default: lasttime
--sortby-direction sortby direction, default: asc
cif -q example.com
2015-11-18 06:16:30,173 - WARNING - cifsdk.client[361] - operation not supported
Needs to be updated for argument change
https://github.com/csirtgadgets/py-cifsdk/blob/master/cifsdk/client.py#L261
cif --otype fqdn -c 85 --feed -f csv
2015-11-18 13:55:31,037 - ERROR - cifsdk.client[368] - unhashable type: 'dict'
cif --otype url -c 85 --feed -f csv
2015-11-18 13:56:55,667 - ERROR - cifsdk.client[368] - global name 'tag_contains_whitelist' is not defined
cif --otype ipv4 -c 85 --feed -f csv
2015-11-18 13:58:10,338 - ERROR - cifsdk.client[368] - 'ascii' codec can't encode characters in position 5-6: ordinal not in range(128)
cif --otype email -c 85 --feed -f csv
2015-11-18 13:58:57,740 - ERROR - cifsdk.client[368] - 'NoneType' object is not callable
using the wrong token surfaces this error:
code:
import time
from cifsdk.client import Client
import json
from pprint import pprint
cli = Client(token='wrong',
remote='https://192.168.1.210',
verify_ssl=False)
ret = cli.search('example.com')
pprint(ret)
error:
$ python wf-submit.py
No handlers could be found for logger "cifsdk.client"
in the perl sdk we have the following:
--id STRING specify an id to retrieve
in the perl client we do not write an output table when there are zero results. in the python client we do. mirror the perl client?
cif -q example112.com -l 3
-----+----------+------------+-------------+------+-------+----------+
| tlp | reporttime | observable | otype | cc | asn | asn_desc | confidence | description | tags | rdata | provider |
+-----+------------+------------+-------+----+-----+----------+------------+-------------+------+-------+----------+
+-----+------------+------------+-------+----+-----+----------+------------+-------------+------+-------+----------+
the install directions to not say to sudo pip install...
which results in this error:
error: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pytricia.so'
----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_giovino/pytricia/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-X4tgUX-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_giovino/pytricia
Need to update the install instructions with:
sudo pip install git+https://github.com/csirtgadgets/py-cifsdk
python sumbit.py
['{"observable":"example4.com","tlp":"amber","confidence":"85","tags":"malware","provider":"me.com","group":"everyone"}', '{"observable":"example4.com","tlp":"amber","confidence":"85","tags":"malware","provider":"me.com","group":"everyone"}']
No handlers could be found for logger "cifsdk.client"
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "sumbit.py", line 23, in worker
ret = cli.submit(data)
File "/home/giovino/py-submit/venv/local/lib/python2.7/site-packages/cifsdk/client.py", line 130, in submit
self.logger.error(json.loads(body.text).get('message'))
ValueError: Expected object or value
when running the whiltelist_testing.sh script got the following error:
#verify ipv6
tmp=`$CIF_CMD --token $EVERYONE_TOKEN --feed --otype ipv6 -c 85 -f csv | grep '$IPV6'`
if [ $? != 1 ]; then echo "ERROR: $IPV6 does not exist for everyone in ipv6 feed"; fi
Traceback (most recent call last):
File "/usr/local/bin/cif", line 9, in <module>
load_entry_point('cif-sdk-py==2.0.0a2-12-gda15b3d', 'console_scripts', 'cif')()
File "/usr/local/lib/python2.7/dist-packages/cifsdk/client.py", line 407, in main
ret = f().process(ret, wl)
File "/usr/local/lib/python2.7/dist-packages/cifsdk/feed/ipv6.py", line 24, in process
raise RuntimeError('https://github.com/jsommers/pytricia/issues/6')
RuntimeError: https://github.com/jsommers/pytricia/issues/6
perl
--whitelist-confidence=INT by confidence (greater or equal to) (default 25)
README.md says the content is LGPL licensed, but LICENSE says Apache
probably need to store true here:
https://github.com/csirtgadgets/py-cifsdk/blob/master/cifsdk/client.py#L210
probably need to read in "stdin" here:
https://github.com/csirtgadgets/py-cifsdk/blob/master/cifsdk/client.py#L352
and submit "stdin"...
The question is, stdin is a string not a dist or list of dicts, what's the best want to convert that string? In the perl sdk we expect json and json'ify it.
The cli is missing the '--description' argument seen in the perl client.
Following the install instructions results in an ImportError when testing the client:
ImportError: No module named prettytable
Easily fixed with:
sudo pip install PrettyTable
This was on an Ubuntu 12 LTS box.
wes@cif:~$ cif -d --tags phishing --provider apwg --limit 5 --otype url
2015-04-29 11:39:04,127 - DEBUG - cifsdk.client::MainThread - uri: https://localhost/observables
2015-04-29 11:39:04,127 - DEBUG - cifsdk.client::MainThread - params: {"nolog":null,"otype":"url","provider":"apwg","limit":"5","tags":"phishing"}
2015-04-29 11:39:04,127 - INFO - cifsdk.client::MainThread - searching...
2015-04-29 11:39:04,141 - INFO - requests.packages.urllib3.connectionpool::MainThread - Starting new HTTPS connection (1): localhost
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
2015-04-29 11:39:04,167 - ERROR - cifsdk.client::MainThread - connection error
2015-04-29 11:39:04,168 - ERROR - cifsdk.client::MainThread - 'NoneType' object has no attribute 'get'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cifsdk/client.py", line 258, in main
print(Table(ret))
File "/usr/local/lib/python2.7/dist-packages/cifsdk/format/table.py", line 24, in __repr__
y = obs.get(c) or ''
AttributeError: 'NoneType' object has no attribute 'get'
perl cli
-q, --query=STRING specify a search
python cli
-q SEARCH, --search SEARCH
search for observable
we should standardize.. --query or --search
perl
--ttl=INT specify number of pings to send, default: 4
(0 infinite, halt with SIGINT or CTRL+C)
perl
--filename specify output filename [default: STDOUT]
When installing py-cifsdk on a CIF server, py-cifsdk overwrites the perl version of /usr/local/bin/cif.
Is this the expected behavior? Seems not ideal to me although I fully realize then you you have to deal with different names for the cif bin.
-g
perl client
--group=STRING by groups (everyone,group1,group2, ...)
this does not handle non-mulitpart messages
body.append(parse_message_part(msg))
https://github.com/csirtgadgets/py-cifsdk/blob/master/cifsdk/email.py#L136
it has to be something like:
if msg.get_default_type() == "text/plain":
print("get_payload-1:", msg.get_payload(decode=True))
elif msg.get_default_type() == "text/html":
print("get_payload-2:", msg.get_payload(decode=True))
else:
# not sure a non mulitpart message has anything be text/plain and text/html
print("WARNING: unhandled default.type", msg.get_default_type())
This exists in the perl client, not in the python client.
perl
--application=STRING filter based on the application field
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.