endgameinc / youarespecial Goto Github PK

Where we can find the data set repository for benign and malicious applications

can i use the Ember data set directly as input for the malicious and benign data, if so how can I do it, as in which path to give and where?

great code thanks
it become famous for example mentioned in the book
Machine Learning and Security Protecting Systems with Data and Algorithms O’Reilly Media 2018
Only it would be very kind of you to share data "100K malicious and benign samples", as you mention , to reproduce your results

So it will be possible to use this code
def fetch_samples():
''' fetches malicious and benign samples form disk
For simplicity, it assumes that samples are named by their sha256 hash, and
are stored in benign/ and malicious/ subdirectories.
Returns a dict "labels", where
labels[sha256] = 0 for a benign sha256, or
labels[sha256] = 1 for a malicious sha256'''
labels = [(sha256, 0) for sha256 in os.listdir('benign')]
labels += [(sha256, 1) for sha256 in os.listdir('malicious')]
labels = dict(labels)
return labels

I'm getting the following error while running it, what could be the reason? I have printed out the labels FYI

python modeltest_multilayer.py
<TarInfo 'ember' at 0x1fc8ebc52a0>
<TarInfo 'ember/train_features_1.jsonl' at 0x1fc8ebc5430>
<TarInfo 'ember/train_features_0.jsonl' at 0x1fc8ebc54f8>
<TarInfo 'ember/ember_model_2017.txt' at 0x1fc8ebc55c0>
<TarInfo 'ember/train_features_3.jsonl' at 0x1fc8ebc5688>
<TarInfo 'ember/test_features.jsonl' at 0x1fc8ebc5750>
<TarInfo 'ember/train_features_5.jsonl' at 0x1fc8ebc5818>
<TarInfo 'ember/train_features_4.jsonl' at 0x1fc8ebc58e0>
<TarInfo 'ember/train_features_2.jsonl' at 0x1fc8ebc59a8>
{<TarInfo 'ember' at 0x1fc8ebc52a0>: 0, <TarInfo 'ember/train_features_1.jsonl' at 0x1fc8ebc5430>: 0, <TarInfo 'ember/train_features_0.jsonl' at 0x1fc8ebc54f8>: 0, <TarInfo 'ember/ember_model_2017.txt' at 0x1fc8ebc55c0>: 0, <TarInfo 'ember/train_features_3.jsonl' at 0x1fc8ebc5688>: 0, <TarInfo 'ember/test_features.jsonl' at 0x1fc8ebc5750>: 0, <TarInfo 'ember/train_features_5.jsonl' at 0x1fc8ebc5818>: 0, <TarInfo 'ember/train_features_4.jsonl' at 0x1fc8ebc58e0>: 0, <TarInfo 'ember/train_features_2.jsonl' at 0x1fc8ebc5a70>: 0, <TarInfo 'ember' at 0x1fc8ebc5b38>: 1, <TarInfo 'ember/train_features_1.jsonl' at 0x1fc8ebc5c00>: 1, <TarInfo 'ember/train_features_0.jsonl' at 0x1fc8ebc5cc8>: 1, <TarInfo 'ember/ember_model_2017.txt' at 0x1fc8ebc5d90>: 1, <TarInfo 'ember/train_features_3.jsonl' at 0x1fc8ebc5e58>: 1, <TarInfo 'ember/test_features.jsonl' at 0x1fc8ebc5f20>: 1, <TarInfo 'ember/train_features_5.jsonl' at 0x1fc8ebdb048>: 1, <TarInfo 'ember/train_features_4.jsonl' at 0x1fc8ebdb110>: 1, <TarInfo 'ember/train_features_2.jsonl' at 0x1fc8ebdb1d8>: 1}
{} //This is the sample_index and X.dat is 0kB as it is empty
Traceback (most recent call last):
File "modeltest_multilayer.py", line 8, in
X, y, sha256list = common.extract_features_and_persist()
File "C:\Users\srihitha.yadlapalli\Desktop\youarespecial-master\classifier\common.py", line 98, in extract_features_and_persist
len(sample_index), -1)
ValueError: cannot reshape array of size 0 into shape (0,newaxis)