Comments (4)
@charlesfair please look at the following script an see if it, at least in the interim, fits your needs:
https://github.com/aeppert/shell_scripts/blob/master/dump_steno.sh
from stenographer.
Thank you @aeppert for the script! That will be much easier than doing it by hand.
One issue with using stenoread for the extraction is how when it is highly optimized to extract small, concise PCAP exactions. When I use it to export a large amount of PCAP, it basically takes over all the cores on the system where it is running. For instance on my test system, which was a quad-core I7 that was hyperthreading with 16GB RAM, stenoread was using around 700% of the CPU percentage, on all cores. I tried to do a full 24 hour PCAP export so as to not fill up the hard drive, but the stenoread process would effectively keep the Stenographer sensor, and Bro, from reliability capturing data during the export.
Basically it is a really really bad idea to export a large amount of Stenographer data as PCAP on a system that is actively collecting network data.
@dcode was interested if the author of Stenographer could create a utility to simply export massive amounts of PCAP without hammering the system during the export
Reference: issue #162
@gconnell
"In general, Steno is optimized to store lots of packets and return a small, targeted set. Note that if you're returning any appreciable percentage of the packets you store, Steno is probably thrashing your disk VERY hard."
The "small utility" that @dcode is interested in would be optimized specifically to simply export a large set of all packets, between two date/time ranges or such that would not thrash your disk.
from stenographer.
This is exactly what I'm interested in now. With no ability to trim captures in steno, or to cut off elephant flows, data consumption is huge. I'd need 170TB of storage space to store 30 days worth of captures in our environment.
If we can take the oldest capture files and convert them into pcaps so we can run trimpcap on them and then store them off somewhere else, that would be huge.
from stenographer.
from stenographer.
Related Issues (20)
- Ubuntu 18.04 Seccomp Failures HOT 11
- gRPC support for stenoread HOT 7
- [Ubuntu] Stenotype keeps crashing HOT 9
- Prometheus metrics support
- Stenographer will not start on Ubuntu 16.04LTS HOT 1
- Temporary directory created before dropping privileges
- Wishlist: Community ID indexed flow extraction
- VLAN tags HOT 3
- multiple threads writing to same disk causes issues with disk cleanup HOT 10
- No such device exception seen while running install.sh on Ubuntu 18.04 HOT 1
- Export the packet through the stenoread and it will automatically break in 15 minutes HOT 1
- rpm created stenographer doesn't run HOT 3
- Notifying about index completion HOT 1
- Decapsulate ERSPAN
- Watchdog failure stenotype abort after 2 minutes (Debian 10) HOT 15
- BlockFile.AllPackets() fail when using non-standard blocksize
- Not able to set stenotype packet directory HOT 1
- Non-pretty results from output file open failure HOT 3
- PCAP-over-IP client in stenotype
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stenographer.