opensoc / opensoc-streaming

Extensible set of Storm topologies and topology attributes for streaming, enriching, indexing, and storing telemetry in Hadoop.

#Current Build

The latest build of OpenSOC-Streaming is 0.6BETA.

We are still in the process of merging/porting additional features from our production code base into this open source release. This release will be followed by a number of additional beta releases until the port is complete. We will also work on getting additional documentation and user/developer guides to the community as soon as we can. At this time we offer no support for the beta software, but will try to respond to requests as promptly as we can.

OpenSOC-Streaming

Extensible set of Storm topologies and topology attributes for streaming, enriching, indexing, and storing telemetry in Hadoop. General information on OpenSOC is available at http://opensoc.github.io

Documentation

Please see documentation within each individual module for description and usage instructions. Sample topologies are provided under OpenSOC_Topologies to get you started with the framework. We pre-assume knowledge of Hadoop, Storm, Kafka, and HBase.

opensoc-streaming's Issues

Double configuration string in configuration file

When running the topology pcap the (pcap) parser bolt is set enabled/disabled in two configuration files, namely in topology.conf as well as in features_enabled.conf. Both files can be found in OpenSOC-Topologies/src/main/resources/OpenSOC_Configs/topologies/pcap.

Suggestion: Enable/Disable the bolt in features_enabled.conf and configure in topology.conf

kraken-pcap 1.5 is unavailable

http://download.krakenapps.org is down,I can't open the website for download the kraken-pcap 1.5 jar.would you show me another adreess

Wrong configuration string *bolt.parser* confused with *parser.bolt*

When running the topology pcap, enabling or disabling the parser bolt will have no effect. The class
TopologyRunner will ask the configuration for the boolean value of "parser.bolt.enabled" whereas in the configuration file the string "bolt.parser.enabled" is used.

Suggestion: Stick to the strategy in the configuration file. All strings start with the type of the element they are enabling. Change TopologyRunner to ask for "bolt.parser.enabled". The class can be found in OpenSOC-Topologies/src/main/java/com.opensoc.topology.runner.

how to run project?

how to run project?

how to run OpenSOC-Pcap_Service

base the instructor , i can run the topology of com.opensoc.topology.Pcap, and it save into hbase normally , but now i don't konw how to display the pacakets by the way of PcapServce.
Someone can help me ?
Thanks!

ES rotating bolt does not always apply alias

Sometimes the ES rotating bolt does not apply aliases when the indexes are first created

Add enrichment adapter to consume data from MISP (Malware Information Sharing Platform)

MISP (Malware Information Sharing Platform) is a project that allows communities to share IOCs and have a centralized repository.
It is in use by a number of CERTs and private companies.

It has a REST API available, and I can help arrange access to live instances if you want to test.

about the opensoc-streaming project which version of kafka storm and other should be installed accurately

i want to build the environment of the opensoc, so i want to know which version of the kafka storm hadoop hbase and other software should be installed . because , when i installed the newest version of above of it , they could not work in right way, and much issues have been occured . Some one could give me a help ?
thanks a lot

pycapa performance

pycapa performance is too low,100MB packets can only cache 2MB.