casics / nostril Goto Github PK

Make Nostril be available from PyPI and possibly other package hosting sites.

Getting the below error

Traceback (most recent call last):
  File "/usr/bin/nostril", line 25, in <module>
    plac.call(main)
  File "/usr/lib/python3.7/site-packages/plac_core.py", line 330, in call
    cmd, result = parser.consume(arglist)
  File "/usr/lib/python3.7/site-packages/plac_core.py", line 207, in consume
    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
  File "/usr/lib/python3.7/site-packages/nostril/__main__.py", line 83, in main
    analyze(strings, trace)
  File "/usr/lib/python3.7/site-packages/nostril/__main__.py", line 95, in analyze
    'nonsense' if nonsense(s) else 'real'))
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 906-917: surrogates not allowed

when trying to detect gibberish from user-generated content.

Hello!
I maintain https://github.com/nexB/scancode-toolkit and I have been looking for such a library which looks really nice (my use case is to remove gibberish from the strings collected from binaries). ScanCode is Apache-licensed. Would you consider some other licensing for nostril, such as any of a classpath-like exception to the GPL, the LGPL or any other permissive license?
Thank you for you kind consideration!

NB: incidentally, scancode may be of some use in the CASICS project.

I can not work on strings less than 6 characters at all? I mean there is no why to go around the algorithm to test short characters?

Provisionally, the following files look like they have to be edited to change all references to the license terms:

• LICENSE
• nostril/__version__.py
• README.md
• NEWS.md

To be honest, providing input from a file didn't work either, but in my opinion stdin support is of even greater importance in Linux environment.

I would expect this command pipeline to work:
printf "%s\n%s\n" "test" "kldfjgnkdgfjn" | nostril

Take this for example:

31d51a69d5ca0407d279a2ece4b3eacd60610444359a0ad099931a516f163397

It is a SHA-256 hash represented as a hex string.

Common sense says it's gibberish. Yet too often are such hashes reported by nostril as real texts.

Since the time this was written, I've come up with slightly more clear installation instructions for installing from github. The instructions for Nostril should be updated.

It turns out the use of the Caltech seal used on the bottom of the README is discouraged by Caltech's media relationships guidelines. Needs to be switched to the Caltech "C" icon.

It looks like Nostril cannot work with PyInstaller ?

I got a Windows EXE but then I got this error :

File "site-packages\nostril\nonsense_detector.py", line 945, in
File "site-packages\nostril\nonsense_detector.py", line 621, in generate_nonsense_detector
ValueError: Cannot find pickle file C:\Users\user\AppData\Local\Programs\Python\Python37\projects\dn_util\dist\site-packages\nostril\ngram_data.pklz

after which, I copied the file ngram_data.pklz manually to the folder in which it is being sought,
but then another error occured (an Exception when I ran my program)

_[Errno 2] No such file or directory: 'C:\Users\user\AppData\Local\Temp\MEI49123\wordsegment\unigrams.txt'

So I believe, if the program was to work seamlessly with PyInstaller, I probably wouldn't need to touch anything.

Notice that everything works perfectly in the Sublime Text IDE.

Is there a function to return actual score rather than a boolean True/False

Hello,
In the training directory, there's some brief explanation about training.
the README file is pointing to a not existing ngram file (there's no ngram_frequencies file in the repo).
Is it possible to describe a quick procedure how to perform the training to get the final ngram pickle file, and what are the requirements about the corpora file format?
thanks
Fausto

The API documentation should be made more explicit and added to readthedocs.org.

Although py.test did not work because of issues with pickling, perhaps another testing framework such as nose or unittest.

I can’t check phrases from a text file. Perhaps I did not understand how to correctly transfer the file name

nostril -f wordlist.txt
nostril: error: argument -f/--file: invalid NoneType value: 'wordlist.txt'

nostril -f /home/user/nostril/wordlist.txt
nostril: error: argument -f/--file: invalid NoneType value: '/home/nekto/nostril/wordlist.txt

There's a rather large memory leak in this library. I won't have the time to find it, but noticed it by forking (subprocess) a project many times and importing nostril causes the machine's RAM to quickly be used up. Without importing nostril the program runs fine with constant memory.

This is the error I receive upon running the demo. Not sure if this is python version error or not.

from nostril import nonsense
ImportError: cannot import name 'nonsense' from 'nostril'.