Usage data sanitization about resh

[curusarn]

paths -> replace dirs except for common ones
git url -> replace url, replace dirs of path

replace parts of arguments if possible:
scp [email protected]:path/path -> scp HASH@HASH:HASH/HASH

replace whole arguments if they don't match any special form
git commit -m "message" -> git commit -m HASH

keep standard arguments and commands (?)

[curusarn]

Almost done: https://github.com/curusarn/resh/tree/dev_2

[curusarn]

I'm handling different types of data differently.

Types

Single value entries

e.g. username, hostname (usually sensitive information)

1. replace with its hash
   • no exceptions, no whitelist

Paths

1. split by /
2. repace each part by its hash
   • unless it's in the whitelist or it's only one character
3. append together

Git origin URL

1. parse the URL using this library https://github.com/whilp/git-urls
2. repace each part by its hash
   • unless it's in the whitelist or it's only one character long
3. get a string of sanitized URL

Command line

I need to replace the command and arguments separately so that I can analyze partial matches later.
However, I don't want to parse bash.
I'm doing the following:

1. split the line into consecutive strings of letters and/or digits (tokens)
   • command options are detected and left unhashed
2. replace each token with its hash
   • unless it's in the whitelist or it's only one character long
3. append together

Whitelisting

I created a whitelist containing various common strings.

• directories in /
• commands installed by default on Ubuntu, Debian or Fedora
• bash and zsh keywords and builtins
• file-extensions
• git subcommands
• some more stuff added by hand:
  • "com", "cz", ...
  • "vim", "emacs", ...
  • "Makefile", "Dockerfile", ...
  • ...

TL;DR

I pretty much hash everything except:

• Commandline options
• All non-alphanumeric chars
• Single-letter or single-digit strings
• Anything whitelisted (see above^)

[curusarn]

• TODO: add more file extensions to the whitelist

• TODO: show this to people

[curusarn]

I have shown this to 3 of my colleagues. Everyone was okay with the result. I got a suggestion that data is sanitized too much.

[curusarn]

I have found a couple of file extension databases. However, they don't seem very fond of other people using their data. I have asked FileInfo.com for permission to use their data.

• WAIT for fileinfo.com to message you back

[curusarn]

I have added a few of common TLDs to the list. Source: https://www.hayksaakian.com/most-popular-tlds/