sidtechnical / hakuna-metadata-1

Browsing metadata visualization tool: Explore how ISPs can see and monetize your browsing history.

Hakuna Metadata - 1

Browsing history visualization for (Linux or Apple) + Firefox combo.

P.S: Read the blog post before trying this tool.

Install instructions

For Linux

• Download and unzip this repository.
• In the unzipped folder on your computer, right click on an empty space and select "Open in terminal" option.
• In the terminal, type sh requirements and press Enter.
• The above command will install all the dependencies.
• In the terminal, type python tool.py and press Enter.
• Be patient, it might take a while. Once the program execution is done, it automatically opens up a web browser tab and displays the visualization.

For Apple macOS

• Download and unzip this repository.
• If you don't have Java installed, install it from here
• In the unzipped folder on your computer, right click on an empty space and select "Open in terminal" option.
• In the terminal, type sh requirements-iOS and press Enter.
• The above command will install all the dependencies.
• In the terminal, type python tool-iOS.py and press Enter.
• Be patient, it might take a while. Once the program execution is done, it automatically opens up a web browser tab and displays the visualization.

Screenshots of working example

Browsing Pattern HeatMap

Browsing Pattern Anamoly

Search Keywords

Psuedo Social Circles

To-Do

• Support for Windows and Chrome (or any other browser)
• Improve the psuedo-social sphere by grouping by range rather than absolute value.

hakuna-metadata-1's Issues

lambda error

Hi again,

I followed your instructions from #1 and it worked, they renamed the module, as you say. Thanks for making the effort of debugging :)

shortly after though, another error pops up; as it seems they changed more than just the module name.

I'd understand it completely, if you didn't want to align to the new pandas version and rewrite everything^^ maybe you can just give name the version of pandas with which it works?

$ python tool.py 
################################################################################################
Please be patient. This might take a while. 
When the tool completes running, it automatically opens a browser tab and displays the visuals.

Unless the program stops with errors, ignore the warnings/messages if any.

If you want to interrupt the program execution, please press CTRL+C.
################################################################################################
No handlers could be found for logger "tldextract"
Traceback (most recent call last):
  File "tool.py", line 480, in <module>
    Unique_URLs_df['Unique_URL_Count'] = Unique_URLs_df.apply(lambda row: Count_Unique_URLs(row['Unique_URLs']), axis=1)                                                            
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/series.py", line 2355, in apply                                                                                          
    mapped = lib.map_infer(values, f, convert=convert_dtype)                                                                                                                        
  File "pandas/_libs/src/inference.pyx", line 1569, in pandas._libs.lib.map_infer (pandas/_libs/lib.c:66440)                                                                        
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/series.py", line 2342, in <lambda>                                                                                       
    f = lambda x: func(x, *args, **kwds)                                                                                                                                            
TypeError: <lambda>() got an unexpected keyword argument 'axis'

Doesn´t work on Ubuntu 16.04.4 LTS

python tool.py produces this output:

No handlers could be found for logger "tldextract" Traceback (most recent call last): File "tool.py", line 336, in <module> facebook_df['category_tags']= facebook_df.apply(lambda row: Facebook_URL_Categorizre(row['url_location']), axis=1) File "/usr/local/lib/python2.7/dist-packages/pandas/core/frame.py", line 2357, in __setitem__ self._set_item(key, value) File "/usr/local/lib/python2.7/dist-packages/pandas/core/frame.py", line 2422, in _set_item self._ensure_valid_index(value) File "/usr/local/lib/python2.7/dist-packages/pandas/core/frame.py", line 2404, in _ensure_valid_index raise ValueError('Cannot set a frame with no defined index ' ValueError: Cannot set a frame with no defined index and a value that cannot be converted to a Series

Python PIL (outdated) missing in requirements, use PILLOW

Python PIL is missing in the requirements but seems to be required.

Since PIL is practically dead, it should be replaced by pillow.

Therefore, add: sudo pip install pillow

Requirement missing

Hi :)

I failed to run your script.

Steps to reproduce:

1. sh requirements

$ sh requirements
[sudo] Passwort für b3yond: 
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.       
Statusinformationen werden eingelesen.... Fertig
»python-pip« ist bereits die neuste Version (8.1.1-2ubuntu0.4).
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 1 nicht aktualisiert.
The directory '/home/b3yond/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                               
The directory '/home/b3yond/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                                      
Collecting pandas
  Downloading pandas-0.20.1-cp27-cp27mu-manylinux1_x86_64.whl (22.3MB)
    100% |████████████████████████████████| 22.3MB 56kB/s 
Collecting pytz>=2011k (from pandas)
  Downloading pytz-2017.2-py2.py3-none-any.whl (484kB)
    100% |████████████████████████████████| 491kB 2.2MB/s 
Requirement already satisfied: python-dateutil in /usr/lib/python2.7/dist-packages (from pandas)
Collecting numpy>=1.7.0 (from pandas)
  Downloading numpy-1.12.1-cp27-cp27mu-manylinux1_x86_64.whl (16.5MB)
    100% |████████████████████████████████| 16.5MB 71kB/s 
Installing collected packages: pytz, numpy, pandas
Successfully installed numpy-1.12.1 pandas-0.20.1 pytz-2017.2
The directory '/home/b3yond/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                               
The directory '/home/b3yond/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                                      
Collecting nltk
  Downloading nltk-3.2.2.tar.gz (1.2MB)
    100% |████████████████████████████████| 1.2MB 554kB/s 
Requirement already satisfied: six in /usr/lib/python2.7/dist-packages (from nltk)
Installing collected packages: nltk
  Running setup.py install for nltk ... done
Successfully installed nltk-3.2.2
The directory '/home/b3yond/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                               
The directory '/home/b3yond/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                                      
Collecting plotly
  Downloading plotly-2.0.8.tar.gz (964kB)
    100% |████████████████████████████████| 972kB 680kB/s 
Collecting decorator (from plotly)
  Downloading decorator-4.0.11-py2.py3-none-any.whl
Collecting nbformat>=4.2 (from plotly)
  Downloading nbformat-4.3.0-py2.py3-none-any.whl (154kB)
    100% |████████████████████████████████| 163kB 2.7MB/s 
Requirement already satisfied: pytz in /usr/local/lib/python2.7/dist-packages (from plotly)
Requirement already satisfied: requests in /usr/local/lib/python2.7/dist-packages (from plotly)
Requirement already satisfied: six in /usr/lib/python2.7/dist-packages (from plotly)
Collecting jupyter-core (from nbformat>=4.2->plotly)
  Downloading jupyter_core-4.3.0-py2.py3-none-any.whl (76kB)
    100% |████████████████████████████████| 81kB 1.8MB/s 
Collecting traitlets>=4.1 (from nbformat>=4.2->plotly)
  Downloading traitlets-4.3.2-py2.py3-none-any.whl (74kB)
    100% |████████████████████████████████| 81kB 5.1MB/s 
Collecting ipython-genutils (from nbformat>=4.2->plotly)
  Downloading ipython_genutils-0.2.0-py2.py3-none-any.whl
Collecting jsonschema!=2.5.0,>=2.4 (from nbformat>=4.2->plotly)
  Downloading jsonschema-2.6.0-py2.py3-none-any.whl
Requirement already satisfied: enum34; python_version == "2.7" in /usr/lib/python2.7/dist-packages (from traitlets>=4.1->nbformat>=4.2->plotly)
Collecting functools32; python_version == "2.7" (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2->plotly)
  Downloading functools32-3.2.3-2.zip
Installing collected packages: decorator, ipython-genutils, traitlets, jupyter-core, functools32, jsonschema, nbformat, plotly
  Running setup.py install for functools32 ... done
  Running setup.py install for plotly ... done
Successfully installed decorator-4.0.11 functools32-3.2.3.post2 ipython-genutils-0.2.0 jsonschema-2.6.0 jupyter-core-4.3.0 nbformat-4.3.0 plotly-2.0.8 traitlets-4.3.2
The directory '/home/b3yond/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                               
The directory '/home/b3yond/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                                      
Requirement already satisfied: beautifulsoup4 in /usr/lib/python2.7/dist-packages
The directory '/home/b3yond/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                               
The directory '/home/b3yond/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                                      
Requirement already satisfied: requests in /usr/local/lib/python2.7/dist-packages
The directory '/home/b3yond/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                               
The directory '/home/b3yond/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                                      
Collecting tldextract
  Downloading tldextract-2.0.2-py2.py3-none-any.whl (52kB)
    100% |████████████████████████████████| 61kB 710kB/s 
Requirement already satisfied: idna in /usr/lib/python2.7/dist-packages (from tldextract)
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages (from tldextract)
Collecting requests-file>=1.4 (from tldextract)
  Downloading requests-file-1.4.2.tar.gz
Requirement already satisfied: requests>=2.1.0 in /usr/local/lib/python2.7/dist-packages (from tldextract)
Requirement already satisfied: six in /usr/lib/python2.7/dist-packages (from requests-file>=1.4->tldextract)
Installing collected packages: requests-file, tldextract
  Running setup.py install for requests-file ... done
Successfully installed requests-file-1.4.2 tldextract-2.0.2
The directory '/home/b3yond/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                               
The directory '/home/b3yond/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.                                                                                                                      
Collecting wordcloud
  Downloading wordcloud-1.3.1.tar.gz (169kB)
    100% |████████████████████████████████| 174kB 1.6MB/s 
Collecting matplotlib (from wordcloud)
  Downloading matplotlib-2.0.2-cp27-cp27mu-manylinux1_x86_64.whl (14.5MB)
    100% |████████████████████████████████| 14.5MB 32kB/s 
Requirement already satisfied: numpy>=1.6.1 in /usr/local/lib/python2.7/dist-packages (from wordcloud)
Requirement already satisfied: pillow in /usr/lib/python2.7/dist-packages (from wordcloud)
Collecting cycler>=0.10 (from matplotlib->wordcloud)
  Downloading cycler-0.10.0-py2.py3-none-any.whl
Requirement already satisfied: python-dateutil in /usr/lib/python2.7/dist-packages (from matplotlib->wordcloud)
Requirement already satisfied: functools32 in /usr/local/lib/python2.7/dist-packages (from matplotlib->wordcloud)
Requirement already satisfied: six>=1.10 in /usr/lib/python2.7/dist-packages (from matplotlib->wordcloud)
Requirement already satisfied: pytz in /usr/local/lib/python2.7/dist-packages (from matplotlib->wordcloud)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 in /usr/lib/python2.7/dist-packages (from matplotlib->wordcloud)
Collecting subprocess32 (from matplotlib->wordcloud)
  Downloading subprocess32-3.2.7.tar.gz (54kB)
    100% |████████████████████████████████| 61kB 4.1MB/s 
Installing collected packages: cycler, subprocess32, matplotlib, wordcloud
  Running setup.py install for subprocess32 ... done
  Running setup.py install for wordcloud ... done
Successfully installed cycler-0.10.0 matplotlib-2.0.2 subprocess32-3.2.7 wordcloud-1.3.1
[nltk_data] Downloading package stopwords to /home/b3yond/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.
[nltk_data] Downloading package punkt to /home/b3yond/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.

2. python tool.py

$ python tool.py 
Traceback (most recent call last):
  File "tool.py", line 6, in <module>
    from pandas.tseries.resample import TimeGrouper 
ImportError: No module named resample

Suggestions:

sudo pip install seems to run more smoothly with the sudo -H flag. But maybe this is just for my environment.
pandas.tseries.resample doesn't seem to be downloaded by pip. How did you get the module and can you replace it?