Future request: support base64 encoded hashes about name-that-hash

[bee-san]

I think we'll go with the flag option, it's less likely to cause problems with auto-detecting Base64 :)

[bastiaan85]

I would prefer it to fall back in some way, you could just try/except the base64decode call to ignore a ValueError (still warn about it) and try its luck with the original string. Btw I would advise against read().splitlines() as that needlessly copies the file's entire content into working space twice: the full string from read() and then a list of a string per line from splitlines(), all before the actual content is even used. Instead I would work iteratively, and I would simply strip and skip empty lines here

    for nr, line in enumerate(kwargs["file"], start=1):
        line = line.strip()
        if not line:
            logger.trace(f"skipped empty line nr {nr}")
            continue
        logger.trace(f"line {nr} from file: {line}")
        if kwargs["base64"]:
            logger.trace("decoding as base64")
            line = base64.b64decode(line)
            logger.trace(f"hash is now {line}")
            logger.trace(f"b64 decoded i is {line}")   # no decode()?
        
        output.append(HashObj(line.decode("utf-8"), nth, hash_info))

        logger.trace(output)

Also the line logger.trace(f"b64 decoded i is {line}") seems incorrect and thus superfluous, there is no decode in that block, that happens in the HashObj instantiation below it. Personally I would not open the file in rb but in r here (and set encoding="utf-8" in the click.File), but you may have had your reasons for using binary mode.

[bee-san]

Update: It now works like above, however it prints the Base64 decoded hash (instead of the Base64 encoded hash).

For example:

Input: NWY0ZGNjM2I1YWE3NjVkNjFkODMyN2RlYjg4MmNmOTk=
Prints: 5f4dcc3b5aa765d61d8327deb882cf99

But that should be okay :D

PS: I fixed all the silly repeated UTF-8 encoding stuff!

[bee-san]

Good point! The funny thing is that NTH supports base64 in the API level, but not on the CLI level 😅

[bee-san]

@bastiaan85 this feature only works if every single hash in a file is Base64, the problem is that some hash types can be decoded as Base64 so automated decoding might result in it identifying the hash wrongly. Is that okay? :)

[bee-san]

Thanks for this!!! I like the idea. I'll try this:

• Decode as Base64 (if Base64 is enabled, so it doesn't slow down the entire thing)
• If it fails, no biggie -- use the raw string
• If it succeeds, try to identify it's hash. If that fails, go back to the original string.

We risk sometimes having something succeed as base64 and being identified when it's not base64 at all, but that is probably quite minimal.

Personally I would not open the file in rb but in r here (and set encoding="utf-8" in the click.File), but you may have had your reasons for using binary mode.

I do this purely because a cryptographer once angrily told me that I can't assume every single piece of data I get is UTF-8 as sometimes they use UTF-16 😂

[bastiaan85]

I don't understand your reasoning here. You still assume it's UTF-8, as you hard code that decoding here

        output.append(HashObj(i.decode("utf-8"), nth, hash_info))

my question is not related to hard coding UTF-8 or not, my question is why that decode happens on that line and not implicitly inside the file object click.File generates, by using click.File(... , encoding='utf-8') ?

[bee-san]

encoding='utf-8'

Ah, honestly I'm an idiot. Let me fix that ASAP! :)

Edit: I'm refactoring the classes.

It should be possible to check one hash at a time, and then build another class which handles single vs multiple-hashes. That way it can do the Try; Except logic with Base64 :D