trimstray / htrace.sh

My simple Swiss Army knife for http/https troubleshooting and profiling.

_{Created by trimstray and contributors}

Description

htrace.sh is a shell script for http/https troubleshooting and profiling. It's also a simple wrapper around several open source security tools.

For a more detailed understanding of htrace.sh, its parameters, functions and how it all works, run htrace.sh --examples or see the Wiki.

Preview

How To Use

To install htrace.sh itself:

# Clone this repository
git clone https://github.com/trimstray/htrace.sh

# Go into directory
cd htrace.sh

# Install
sudo ./setup.sh install

# Install dependencies (Debian 8/9, Ubuntu 18.x and MacOS support)
#   - recommend build docker image or install dependencies manually
#   - before init please see what it does and which packages are available on your repository
sudo ./dependencies.sh

# Show examples
htrace.sh --examples

# Run the app
htrace.sh -u https://nmap.org -s -h

• symlink to bin/htrace.sh is placed in /usr/local/bin
• man page is placed in /usr/local/man/man8

or build docker image:

# Clone this repository
git clone https://github.com/trimstray/htrace.sh

# Go into directory and build docker image
cd htrace.sh && build/build.sh

# Run the app
docker run --rm -it --name htrace.sh htrace.sh -u https://nmap.org -s -h

Parameters

Provides the following options:

    htrace.sh v1.1.7

  Usage:

    htrace.sh <option|long-option> [value]

  Examples:

    htrace.sh -u https://example.com -s -h -b
    htrace.sh -u https://example.com --all-scans

  Options:

        --help                                show this message
        --version                             show script version
        --examples                            show script examples

    Standard:

        -u|--url <value>                      set target url with http/https protocol
        -s|--ssl                              show basic ssl server/connection parameters
        -h|--headers                          show response headers
        -b|--body                             show response body
        -M|--req-method <value>               set request method (default: GET)
        -H|--req-header <value>               set request header(s)
        -p|--proxy <value>                    set proxy server (not for external tools)
        -r|--resolve <value>                  resolve the host+port to this address
        -i|--iface <value>                    set network interface (or address)
        -a|--all-scans                        use all external security tools

    Security tools:

        --testssl                             test ssl protocols and ciphers (testssl.sh)
        --observatory                         analyze website headers (mozilla observatory)
        --ssllabs                             deep analysis of the ssl web server (ssllabs)
        --mixed-content                       scan website for non-secure resources (mixed-content-scan)
        --nse                                 scan website and domain with nse library (nmap)
        --waf                                 detect and bypass web application firewalls (wafw00f)
        --dns                                 enumerate subdomains (subfinder) and perform zone transfer
        --http2                               test HTTP/2 (nghttp2)

    Extended:

        --ssl-bin <path>                      set path to the openssl bin
        --ssl-debug                           debug ssl connection
        --cache-bypass <value>                try (proxy) cache bypass
        --user-agent <value>                  set 'User-Agent' header
        --referer <value>                     set 'Referer' header
        --auth <value>                        set authentication method
        --httpv <value>                       set http version
        --tlsv <value>                        set tls version
        --ciph <value>                        set of cryptographic algorithm
        --max-redirects <num>                 set max redirects (default: 10)
        --timeout <num>                       set max timeout (default: 15)
        --hide-src-ip                         hide source ip from output

Contributing

See this.

Code Contributors

This project exists thanks to all the people who contribute.

License

GPLv3 : http://www.gnu.org/licenses/

Free software, Yeah!

htrace.sh's Issues

Set cookie or auth header.

A lot of endpoints are behind authentication. The --req-headers flag is feasible, but there should be a shortcut flag just for authentication headers. Maybe --cookie and --auth

Show response body.

Example:

htrace.sh -d https://badssl.com --body

Bypassing Reverse Proxy, Cache and Content Delivery Networks.

PATH env is hardcoded causing issues when external tools are not located in default locations

root@bethebeast:/usr/share/backgrounds# export PATH=/root/yaes:/root/.config/composer/vendor/bin/:/root/infosec/tools:/root/go/bin/:/usr/local/go/bin:$PATH
root@bethebeast:/usr/share/backgrounds# htrace.sh
not found in PATH: ssllabs-scan mixed-content-scan
root@bethebeast:/usr/share/backgrounds# which observatory
/usr/bin/observatory
root@bethebeast:/usr/share/backgrounds# which ssllabs-scan
/root/infosec/tools/ssllabs-scan
root@bethebeast:/usr/share/backgrounds# ln -s /root/infosec/tools/ssllabs-scan /usr/bin/
root@bethebeast:/usr/share/backgrounds# which ssllabs-scan
/root/infosec/tools/ssllabs-scan
root@bethebeast:/usr/share/backgrounds# htrace.sh
not found in PATH: mixed-content-scan

I did not looked into code, but seems like htrace.sh is not looking into $PATH from environment variables?

Reduce the curl command calls.

fgrep _curl_base * -R                                                                                                                                                                                                                 [10:49:38]
lib/DomainResolve:  _host_ip=$($_curl_base -ks -m "$_timeout" "https://dns.google.com/resolve?name=${_host}&type=A" | \
lib/DomainTrace:  _http_output=$($_curl_base -ks -m "$_timeout" \
lib/DomainTrace:    local _via_ip=$($_curl_base_remote -ks -m "$_timeout" http://whatismyip.akamai.com/)
lib/DomainTrace:        $_curl_base -Iks -m "$_timeout" \
lib/DomainTrace:        $_curl_base -Iks -m "$_timeout" \
lib/DomainTrace:        $_curl_base -ks -m "$_timeout" \
lib/DomainTrace:        $_curl_base -ks -m "$_timeout" \
src/__init__:  local _curl_base=""
src/__init__:  local _curl_base_remote=""
src/__init__:    _curl_base="curl --proxy $proxy_type --request $req_method_type"
src/__init__:    _curl_base_remote="curl --proxy $proxy_type"
src/__init__:    _curl_base="curl --request $req_method_type"
src/__init__:    _curl_base_remote="curl"

Too old nmap version in Docker image

Hi again, it's your beta tester number one... ;)

Running on standalone machine with nmap 7.7 it works fine, but for Docker image

 Scan domain with Nmap NSE Library (https://nmap.org/book/nse.html)

         › Starting Nmap 7.40 ( https://nmap.org ) at 2018-08-09 11:43 UTC
         › NSE: failed to initialize the script engine:
         › /usr/bin/../share/nmap/nse_main.lua:816: 'http-cookie-flags' did not match a category, filename, or directory
         › stack traceback:
         ›      [C]: in function 'error'
         ›      /usr/bin/../share/nmap/nse_main.lua:816: in local 'get_chosen_scripts'
         ›      /usr/bin/../share/nmap/nse_main.lua:1271: in main chunk
         ›      [C]: in ?
         › QUITTING!
         › Starting Nmap 7.40 ( https://nmap.org ) at 2018-08-09 11:43 UTC
         › NSE: failed to initialize the script engine:
         › /usr/bin/../share/nmap/nse_main.lua:816: 'http-cookie-flags' did not match a category, filename, or directory
         › stack traceback:
         ›      [C]: in function 'error'
         ›      /usr/bin/../share/nmap/nse_main.lua:816: in local 'get_chosen_scripts'
         ›      /usr/bin/../share/nmap/nse_main.lua:1271: in main chunk
         ›      [C]: in ?
         › QUITTING!
         › Starting Nmap 7.40 ( https://nmap.org ) at 2018-08-09 11:43 UTC
         › NSE: failed to initialize the script engine:
         › /usr/bin/../share/nmap/nse_main.lua:816: 'http-cookie-flags' did not match a category, filename, or directory
         › stack traceback:
         ›      [C]: in function 'error'
         ›      /usr/bin/../share/nmap/nse_main.lua:816: in local 'get_chosen_scripts'
         ›      /usr/bin/../share/nmap/nse_main.lua:1271: in main chunk
         ›      [C]: in ?
         › QUITTING!
         › Starting Nmap 7.40 ( https://nmap.org ) at 2018-08-09 11:43 UTC
         › NSE: failed to initialize the script engine:
         › /usr/bin/../share/nmap/nse_main.lua:816: 'http-jsonp-detection' did not match a category, filename, or directory
         › stack traceback:
         ›      [C]: in function 'error'
         ›      /usr/bin/../share/nmap/nse_main.lua:816: in local 'get_chosen_scripts'
         ›      /usr/bin/../share/nmap/nse_main.lua:1271: in main chunk
         ›      [C]: in ?
         › QUITTING!
         › Starting Nmap 7.40 ( https://nmap.org ) at 2018-08-09 11:43 UTC
         › NSE: failed to initialize the script engine:
         › /usr/bin/../share/nmap/nse_main.lua:816: 'http-cookie-flags' did not match a category, filename, or directory
         › stack traceback:
         ›      [C]: in function 'error'
         ›      /usr/bin/../share/nmap/nse_main.lua:816: in local 'get_chosen_scripts'
         ›      /usr/bin/../share/nmap/nse_main.lua:1271: in main chunk
         ›      [C]: in ?
         › QUITTING!

possible to include newer nmap version in Docker image? possible solution below to include latest nmap

RUN apt update
RUN apt -y install wget
RUN wget https://nmap.org/dist/nmap-7.70-1.x86_64.rpm
RUN apt -y install alien
RUN alien nmap-7.70-1.x86_64.rpm
RUN dpkg -i nmap_7.70-2_amd64.deb
RUN nmap -version

Hide source IP address.

Standard output:

htrace.sh -d https://badssl.com                 

     htrace.sh v1.0.9  (openssl 1.1.0h: ok)


    URI: https://badssl.com

    req  time_total   time_connect    local_socket           via              remote_socket         geo   proto   ver   code     next_hop
    ---  ----------   ------------    ------------           ---              -------------         ---   -----   ---   ----     --------
 •   1   0.657482     0.657482        10.245.203.25:52236    211.105.75.25    104.154.89.105:443    US    https   1.1   200

with --hide-src-ip param:

htrace.sh -d https://badssl.com                 

     htrace.sh v1.0.9  (openssl 1.1.0h: ok)


    URI: https://badssl.com

    req  time_total   time_connect    local_socket           via              remote_socket         geo   proto   ver   code     next_hop
    ---  ----------   ------------    ------------           ---              -------------         ---   -----   ---   ----     --------
 •   1   0.657482     0.657482        xxx.xxx.xxx.xxx:52236  xxx.xxx.xxx.xxx  104.154.89.105:443    US    https   1.1   200

Identify and fingerprint Web Application Firewall (WAF).

The best external tool: wafw00f

Package for distros

It would be good to package this for distros. So I can just do "apt/dnf/… install" and I have it.

Maybe we can use this issue to try to make this happen?

Value for POST method.

./bin/htrace.sh -d https://example.com -M "POST:aaa bbb ccc"

Generate html output report.

• hah
• ansi2html

Added GeoIP (resolves geographic information about an IP address).

Resolves geographic information about an IP address.

geoip-bin -> geoiplookup

Remove all url arguments.

Example:

htrace.sh -d https://nmap.org?test-arg -s

    htrace.sh v1.0.7


    URI: https://nmap.org?test-arg

    req  time_total   time_connect    local_socket           via              remote_socket        type    http  code     next_hop
    ---  ----------   ------------    ------------           ---              -------------        ----    ----  ----     --------
 ›   1   0.702265     0.702265        10.217.11.10:42558     35.230.xxx.xxx   45.33.49.119:443     https   1.1   200      
         ssl: on, version(), cipher()
unable to load certificate
140111198290112:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
unable to load certificate
139785700073664:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
unable to load certificate
140045602996416:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
unable to load certificate
139713580265664:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
unable to load certificate
139865397174464:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
unable to load certificate
140218849345728:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
unable to load certificate
140641044259008:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
         public-key(), signature()
         date:  / 
         issuer: <empty>
         owner: <empty>
         cn: <empty>
         san: <empty>
         validity: is not valid (hostname mismatch)
         verification:

Docker Alpine: geoip empty output

docker run --rm -it --name htrace.sh htrace.sh -u https://nmap.org -h --hide-src-ip

     htrace.sh v1.1.1  (openssl 1.1.1a: not tested)


    URI: https://nmap.org ; Method: GET

    req  time_total   time_connect    local_socket           via              remote_socket         geo   proto   ver   code     next_hop
    ---  ----------   ------------    ------------           ---              -------------         ---   -----   ---   ----     --------
 •   1   1.062680     1.062680        xxx.xxx.xxx.xxx:44328  xxx.xxx.xxx.xxx  45.33.49.119:443            https   1.1   200

Different results between "docker run" and standalone tool

After building Docker image

Docker version 18.06.0-ce, build 0ffa825 ( same happens on Linux machine though )
Darwin Kernel Version 17.7.0:

docker run --rm -e TERM=screen web-security:htrace --domain https://wp.pl/ --mixed-content

    htrace.sh v1.0.6


    URI: https://wp.pl/

    req  time_total   time_connect    local_socket           via              remote_socket        type    http  code     next_hop
    ---  ----------   ------------    ------------           ---              -------------        ----    ----  ----     --------
 ›   1   1.648620     1.648620        172.17.0.2:33722       213.241.3.97     212.77.98.9:443      https   2.0   301      https://www.wp.pl/
 ›   2   1.875227     0.226607        172.17.0.2:33730       213.241.3.97     212.77.98.9:443      https   2.0   200

    Scan domain for Mixed Content (https://github.com/bramus/mixed-content-scan)

         › Mixed content not found

meanwhile standalone machine

root@bethebeast:/usr/share/backgrounds# htrace.sh --domain https://wp.pl --mixed-content

    htrace.sh v1.0.6


    URI: https://wp.pl

    req  time_total   time_connect    local_socket           via              remote_socket        type    http  code     next_hop
    ---  ----------   ------------    ------------           ---              -------------        ----    ----  ----     --------
 ›   1   2.802012     2.802012        172.16.219.169:42066   213.241.3.97     212.77.98.9:443      https   2.0   301      https://www.wp.pl/
 ›   2   5.139284     2.337272        172.16.219.169:42074   213.241.3.97     212.77.98.9:443      https   2.0   200

    Scan domain for Mixed Content (https://github.com/bramus/mixed-content-scan)

         › [2018-08-09 11:53:45] MCS.ERROR: 00000 - https://www.wp.pl/
         › [2018-08-09 11:53:45] MCS.WARNING: http://www.google.pl
         › [2018-08-09 11:53:45] MCS.WARNING: http://www.google.com
         › [2018-08-09 11:53:45] MCS.WARNING: http://www.facebook.com
         › [2018-08-09 11:53:45] MCS.WARNING: http://sportowefakty.wp.pl
         › [2018-08-09 11:53:45] MCS.WARNING: http://wiadomosci.wp.pl
         › [2018-08-09 11:53:45] MCS.WARNING: http://www.money.pl

any idea what may be causing that?

EDIT: on my external VPS I do not have this issue

docker -v
Docker version 18.06.0-ce, build 0ffa825
4.15.0-30-generic #32~16.04.1-Ubuntu SMP

may be smth up with network here...

Add User Agents list.

Randomize _user_agent variable.

Unclear how to create man page

Please delete -- posted on the wrong thing.

Not working on MacOS/Darwin

$ sudo ./setup.sh install
Password:
readlink: illegal option -- f
usage: readlink [-n] [file ...]
Create symbolic link to /usr/local/bin
ln: /usr/local/bin/htrace.sh: File exists
Create man page to /usr/local/man/man8
$ ln -s ~/src/github.com/trimstray/htrace.sh/bin/htrace.sh /usr/local/bin/
$ htrace.sh -d http://nmap.org -s -h
readlink: illegal option -- f
usage: readlink [-n] [file ...]
/usr/local/bin/htrace.sh: line 55: ./../src/settings: No such file or directory
/usr/local/bin/htrace.sh: line 57: ./../src/helpers: No such file or directory
/usr/local/bin/htrace.sh: line 59: ./../src/__init__: No such file or directory
/usr/local/bin/htrace.sh: line 66: __main__: command not found
/usr/local/bin/htrace.sh: line 68: _exit_: command not found
$ bash --version
GNU bash, version 5.0.0(1)-release (x86_64-apple-darwin18.2.0)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Error Kali Linux

When install in Kali Linux:

sudo ./dependencies.sh Autoinstaller is not available on your system.

GeoIP: set null for internal IPs

Example:

    req  time_total   time_connect    local_socket           via              remote_socket         geo   proto   ver   code     next_hop
    ---  ----------   ------------    ------------           ---              -------------         ---   -----   ---   ----     --------
 ›   1   0.011561     0.011561        172.32.220.172:39858   <int_link>       172.32.220.240:443           https   2.0   502

dig: connection timed out

Replace connection timed out output error.

Security scan with Nmap NSE Library.

Security scan with Nmap NSE Library:

local _nmap_nse_scripts="http-auth-finder,\
                         http-chrono,\
                         http-cookie-flags,\
                         http-cors,\
                         http-cross-domain-policy,\
                         http-csrf,\
                         http-dombased-xss,\
                         http-git,\
                         http-grep,\
                         http-internal-ip-disclosure,\
                         http-jsonp-detection,\
                         http-malware-host,\
                         http-methods,\
                         http-passwd,\
                         http-phpself-xss,\
                         http-php-version,\
                         http-robots.txt,\
                         http-sitemap-generator,\
                         http-shellshock,\
                         http-stored-xss,\
                         http-unsafe-output-escaping,\
                         http-useragent-tester,\
                         http-vhosts,\
                         http-xssed,\
                         ssl-enum-ciphers,\
                         whois-ip"

Empty local port number for local_socket

Only when --hide-src-ip is used:

    req  time_total   time_connect    local_socket           via              remote_socket         geo   proto   ver   code     next_hop
    ---  ----------   ------------    ------------           ---              -------------         ---   -----   ---   ----     --------
 •   1   0.266118     0.266118        xxx.xxx.xxx.xxx        xxx.xxx.xxx.xxx  35.228.233.78:443     US    https   1.1   200

Better parsing 'issuer' from certificate.

Better parsing 'issuer' from certificate, eg.:

issuer: DOT Certification Authority, e-mail: [email protected]

./setup.sh install - with dependencies.

Add --with-dependencies param to setup.sh.

• curl
• openssl
• dig
• bc
• observatory
• ssllabs-scan
• mixed-content-scan
• nmap

Add -H|--header for request header(s).

Add -H|--header for request header(s).

Testing SSL protocols and ciphers with testssl.sh

Testing SSL protocols and ciphers with testssl.sh.

Test http/https connection without redirects.

Test http/https connection without redirects, e.g.:

htrace.sh -d https://example.com --without-redirects

Can I do traceroute like functionality without admin privileges?

I need to do something like traceroute/tracepath/tracert, but I have no admin privileges and I do not have traceroute/tracepath/tracert installed. All I have is Java, Python, cygwin, and Windows 7, and I can put things in my home directory but I cannot do full installations. Do you have or know of a way to do traceroute/tracepath/tracert given these limitations?

Docker image for htrace.sh.

Docker automated build image.

Added ssl owner 'Organization' and 'OrganizationalUnit'.

• _ssl_domain_subject_o
• _ssl_domain_subject_ou

Multi threaded tasks.

For all jobs (especially for params).

OpenSSL versions.

• openssl 1.1.0j
• openssl 1.0.2g

tput: No value for $TERM and no -T specified in Docker htrace

Hello,

I build a container with the Docker file but when I try to run it
docker run --rm htrace --domain https://domaine.com

I have the error :
tput: No value for $TERM and no -T specified

Where is my mistake ?

Regards

Cannot build a Docker image by `Unsupported system version.`

As How to use says, I ran cd htrace.sh/build && docker build --rm -t htrace.sh -f Dockerfile . and I got a failure with Unsupported system version..

It seems the failure happens on the line 96 of Dockerfile, which executes setup.sh

if [[ "$OSTYPE" == "darwin"* ]] ; then
  [ ! -z "$(brew --prefix)" ] && PATH=$(brew --prefix)/opt/coreutils/libexec/gnubin:$PATH
  readonly _dir=$(dirname "$(readlink "$0" || echo "$(echo "$0" | sed -e 's,\\,/,g')")")
elif [[ "$OSTYPE" == "linux-gnu" ]] ; then
  readonly _dir=$(dirname "$(readlink -f "$0" || echo "$(echo "$0" | sed -e 's,\\,/,g')")")
else
  printf "Unsupported system version.\\n"
  exit 1
fi

With the Dockerfile, this shell script is executed in alpine:latest image, and $OSTYPE returns linux-musl.

A little problem with the output

Hey there,

As part of the output , I'm getting this before the rest of the output

curl: unknown --write-out variable: 'http_version'
curl: unknown --write-out variable: 'scheme'

What does this imply ? Please check out the attached photo

Wiki: updated 'Examples' chapter

Wiki Examples chapter.

• - Check redirects
• - POST method

Checks common name valid.

Checks common name valid:

• the certificate is valid only if the request hostname matches the certificate common name
• ssl certificate name mismatch error

Set proxy (as a param).

For all traffic, e.g. curl ssllabs, mozilla-observatory.

For nmap:

• https://www.aldeid.com/wiki/Tor/Usage/Nmap-scan-through-tor
• https://www.shellhacks.com/anonymous-port-scanning-nmap-tor-proxychains/

SNI and non-SNI support

Test SNI and non-SNI for TLS.

Automatic redirect loop detection.

Prevent too many redirects.

Unknown gateway (via) with POST method.

htrace.sh -d https://badssl.com -m POST

     htrace.sh v1.0.9  (openssl 1.1.0h: ok)


    URI: https://badssl.com

    req  time_total   time_connect    local_socket           via              remote_socket         geo   proto   ver   code     next_hop
    ---  ----------   ------------    ------------           ---              -------------         ---   -----   ---   ----     --------
 •   1   0.248689     0.248689        192.168.220.30:41780   unknown          104.154.89.105:443    US    https   1.1   405

New external tool: mixed-content-scan

Repo: mixed-content-scan

With new htrace.sh param: --mixed-content.

Incorrect parse url arguments.

• ?q=foo -> URI: <scheme>://<domain>

Set HTTP method.

Example: -m|--method <GET|POST>

Homebrew installation

This looks cool--I saw it mentioned on Hacker News.

I'd like to ask for a Homebrew formula to be created so that I could install this on OS/X with brew install htrace or similar.

If you're not on a Mac or don't want to do this, I'll be happy to take care of it some evening, just let me know!

-- Doug

Reorganization htrace.sh parameters.

• -h|--headers
• -b|--body
• -M|--method
• -H|--header

"not found in PATH" error message.

after run :
htrace.sh --domain https://www.test.com
write this error :
not found in PATH: geoiplookup observatory ssllabs-scan mixed-content-scan

macOS: readlink: illegal option -- f

Both setup.sh and htrace.sh show the error:

readlink: illegal option -- f

The easiest solution is probably to ask macOS users to do brew install coreutils and replace all readlink calls with greadlink if it exists.

Improve 'issuer' and 'owner' metadata.

• add/update o and cn for issuer
• add/update o and ou for owner