kismetwireless / python-kismet-db Goto Github PK
View Code? Open in Web Editor NEWKismetdb database log helper library (mirrored from Kismet repos)
License: GNU General Public License v2.0
Kismetdb database log helper library (mirrored from Kismet repos)
License: GNU General Public License v2.0
I received the message exported 0 devices
after running kismet_log_to_xml
. I confirmed the .kismet file did have device records.
I added more verbose error logging to the script and turns out there's an issue dereferencing the lon/lat data. It looks like Kismet has changed the data structure to a geopoint instead of individual lon/lat fields. Here's an example:
"kismet.common.location.avg_loc": {
"kismet.common.location.time_usec": 0,
"kismet.common.location.time_sec": 0,
"kismet.common.location.fix": 0,
"kismet.common.location.alt": 0,
"kismet.common.location.geopoint": [0, 0]
}
I captured this data on Kismet 2020-12-R3.
Is there a way to attach to a running kismet session? I know I could try to open the .kismet file it generates, but I was wondering if there was a way to use the kismetdb against the kismet session like how the REST API's work?
Improve timestamp handling:
Add a synthetic column for handling timestamps. This will add a column to output which will be a python datetime
object (composed from ts_sec and ts_usec).
Provide functionality to make time-related queries using a Python datetime
object instead of having to reduce query args to two timestamp-related fields.
Add the ability to query using ISO 8601-formatted timestamp. Prioritize so that if multiple timestamps are provided with kwargs, the priority (in order of importance) should be:
I have a python script that pulls out data from a generated kismet file so that the information is presented in a summarised format. This kismet file is generated when you stop running kismet and has an extension of .kismet. This kismet file is actually a SQLite database underneath. However I have an issue whereby when I try to run my python script on a kismet file generated on newer versions of kismet I get quite a few errors, whereas If I run on an older version of kismet I was using before it works fine. I should also point out that I updated all my other libraries and software on my system too. My python script makes use of the kismetdb python wrapper library and a few others to accomplish this extraction of SQLite data.
I have done some investigating and tested a few things. Firstly I checked the version of SQLite that I had used before and what I had now. The one I was using before was version SQLite 3027002 and the one I’m using now is SQLite 3038002. I tested both versions of SQLite with the new version of kismet and it made no difference.
Next I tested different versions of python. My original python version was python 3.7.3 and my new one is now python 3.9.2. I ran my script with both python versions on the new kismet version. On python 3.7.3 I got a different error that says KeyError: 8. Whereas with python 3.9.2 I got a UnicodeDecodeError which I've posted the full error log after my code snippet.
The new kismet version I’m using now is 2022-01-R3 and the one I was using before was 2020-12-R3. Looking at the changelog didn’t really give me an information that I found useful. I also looked at the version of the actual kismet db_versions on both versions of kismet. The newer kismet uses version 8 whereas the older kismet is using version 6 and according to this information version 8 introduces the hash and packetid attributes to the packets table within the generated kismet file.
This is my python script code, I’ve commented out most of it so I can narrow down what’s causing the error on the new kismet version. I’ve managed to determine that it’s when I call the get_all()
function that the error occurs. When I run this code with the old kismet version the print(KIS_DEVICES.get_all(**query_args))
outputs all of the devices that kismet detected, which corresponds to the devices table in the generated kismet SQLite file, which is too long to show. But that’s what I’m trying to do but with the new kismet version that uses kismetdb version 8.
import json, sys, kismetdb
from datetime import datetime
# Check if KismetDB is Specified as an Argument
if not len(sys.argv) == 2:
print("[!] No KismetDB Specified")
sys.exit(0)
# Set Input and Output Files
KIS_IN = sys.argv[1]
KIS_OUT = "%ssummary" % (KIS_IN[:-6])
query_args = {}
# Get Kismet Devices from DB
KIS_DEVICES = kismetdb.Devices(KIS_IN)
print(KIS_DEVICES.get_all(**query_args))
KIS_DB = [row["device"] for row in KIS_DEVICES.get_all(**query_args)]
sys.exit(0)
This is the error I get when I try to run that same code with the newer kismet version:
sudo python3 KismetDB_to_Summary\ copy.py new\ pi\ build/Kismet-20220411-23-01-06-1.kismet
Traceback (most recent call last):
File "/Users/user/Desktop/Apolloo /KismetDB_to_Summary copy.py", line 53, in <module>
print(KIS_DEVICES.get_all(**query_args))
File "/usr/local/lib/python3.9/site-packages/kismetdb/base_interface.py", line 155, in get_all
return self.get_rows(self.column_names, sql, replacements)
File "/usr/local/lib/python3.9/site-packages/kismetdb/base_interface.py", line 325, in get_rows
for row in cur.fetchall():
File "/usr/local/lib/python3.9/site-packages/kismetdb/utility.py", line 473, in device_field_parser
retval = json.dumps(json.loads(device))
File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 341, in loads
s = s.decode(detect_encoding(s), 'surrogatepass')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf5 in position 1286: invalid start byte
I’ve tried looking through the library code that the errors point to but most of it goes over my head and I’m really not sure what could be causing the errors. I have a feeling it’s something to do with the fact that the newer version of kismetdb (version 8) introduces new attributes in the packets table, but I’m not actually sure if that what the issue is.
Any help would be appeciated as I really want to use a newer version of kismet but still be able to extract and filter the generated SQLite data into my summary output.
Getting back a buffer object instead of a string for the json field.
When taking a large Kismet sqlite3 database and using kismet_log_to_pcap, a typeerror is returned:
# kismet_log_to_pcap --in Kismet-20190904-10-18-58-1.kismet --outtitle small --limit-packets 100000
Limiting to 100000 packets per file in small-X.pcap
DLT 0 for all packets
Logging to None
Traceback (most recent call last):
File "/usr/local/bin/kismet_log_to_pcap", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/kismetdb/scripts/log_to_pcap.py", line 110, in main
logf = open(results.outfile, file_mode)
TypeError: coercing to Unicode: need string or buffer, NoneType found
The kismet database has plenty of packets:
# ls -la | grep Kismet
-rw-r--r-- 1 root root 196722688 Sep 4 10:36 Kismet-20190904-10-18-58-1.kismet
Installed on Kali per instructions at https://www.kismetwireless.net/docs/readme/kismetdb_to_pcap/ which references the kismetwireless.net git repo as opposed to this one.
at a minimum, packets.py doesn't support new columns from v7 and v8
v7 added column "datarate"
v8 added columns "hash" and "packetid"
i added an entry for each value in column_reference of packets.py and seems to work, not sure if more is needed
edit: i'll try to submit a pull request with some fixes
The output file of kismet_log_devices_to_json is a correct json file but it is a list of strings, which is not usefull.
It seems each object of the list has been converted to a string by out of placed quote at the start and end of each line of the file, and all the quotes inside the "object" are escaped with \.
For example:
[
"{<object1>}",
"{<object2>}",
"{<object3>}"
]
To get a usefull json file, one can use:
sed -e 's/\\"/"/g' -e 's/ "{/ {/g' -e 's/}",$/},/g' -e 's/}"$/}/g' broken.json > fixed.json
The correct output should have been:
[
{<object1>},
{<object2>},
{<object3>}
]
where each object has quote not escaped.
Tested with a KismetDB version: 6
Also increment version and test Github's issue-to-commit tracking with new process.
Whenever I run kismet_log_to_pcap --in my_rowless_file.kismet --out output.pcap
, the output file is not actually created.
This is likely currently intentional, but I'm not sure that is the right default behavior to have. This was causing issues in my script where I was trying to read data from a file I expected to exist, even if the file was just empty. I've implemented a workaround for the time being, but I really think a solution like this should be the library's default:
filename = await self.convert_kismet_to_pcap(kismet_in=filename)
with open(csv_filename) as read_obj:
extra_info_kismet_df = pd.read_csv(read_obj, low_memory=False, delimiter='\t')
# Handle cases where there is no generated file (because the Kismet file has no rows)
if not os.path.isfile(filename):
with open(filename, "w") as empty_file:
empty_file.write("")
Need to improve docs on included scripts.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.