Version: 2.1.2-20190305
- Privacy International's data interception environment
- Acknowledgements
For those who just want to get analysing
- VirtualBox 6.x Installed
- A Wifi USB Dongle (specifically https://amzn.com/B00JZFT3VS or any other Ralink 5370 Chipset USB Adapter)
- Internet Connectivity
- Some experience of Unix like operating systems (particularly systemd)
- Some experience with Android and the Android Developer Bridge (ADB)
- An Android Handset that is rooted and has ADB enabled. Running Oreo or earlier
Head to Privacy International's website and download the VM: https://privacyinternational.org/mitmproxy19
Download the Virtual Appliance (OVA, extension .ova)
Double Click on the OVA. This should open VirtualBox if .ova is associated with VirtualBox. Otherwise you can click Import Appliance from the File Menu in VirtualBox
Click import
Read and accept the licence agreement
Plug in your Wifi USB if you haven't done so already.
Make sure the Wifi USB dongle is connected to the Virtual Machine ( Menu Bar - Device -> USB )
Start the Virtual machine
Once the desktop has loaded, you will need to run change_wireless_interface.sh and provide it with the device identifier for your wireless network dongle
Open the Terminal (Applications -> System Tools -> QTerminal)
Type the following command sudo service systemctl enable hostapd this will enable the Wireless Daemon
Reboot the Virtual Machine
Check on your mobile device and see if it can connect to the WiFi network "pi_maninthemiddle".
Troubleshooting: (most of these require the use of the terminal)
If you cannot see the "pi_maninthemiddle" network, something is likely wrong with either your hardware or hostapd.
- Double check that you WiFi dongle is connected to the Virtual Machine.
- Make sure the interface name is correct in
sudo nano /etc/hostapd/hostapd.conf - You can run hostapd manually, to see if there is any other issues with starting. e.g:
sudo killall hostapd; sleep 4;sudo hostapd -p /etc/hostapd/hostapd.conf
If you are not being assigned an IP address (Your device can see the network, but it keeps looping between connecting states)
- Restart dnsmasq
sudo service dnsmasq restart
Your device says "Connected, no Internet"
This is good! This means that your device is waiting for mitmproxy to be started
You can start mitmproxy using the "mitmproxy_start.sh" on the desktop
When you have finished with you can end your session "mitmproxy_stop.sh", your data will be saved in home
Once you have started mitmproxy, visit http://mitm.it you should see prompts to install the certificate for your device. Install the certificate, and using an app like Chrome, see if you can browse the Internet
If you are getting errors in the logfile for mitmproxy stating that the certificate is not trusted, see below - Step 6
Google changed the way the certificate store works in Android Nougat, please follow this guide to correctly install your certificate.
As this is a quick start, rough guide is: Get the ID of the Certificate in a format that Android Expects
sudo openssl x509 -inform PEM -subject_hash_old -in /root/.mitmproxy/mitmproxy-ca-cert.pem | head -1
Note the output: e.g "abcdef12"
Copy the Certificate to the Device over ADB
adb push /root/.mitmproxy/mitmproxy-ca-cert.pem /sdcard/<NamefromOpenSSLOutput>.0
Remount the /system partition, copy the certificate, set its permissions (requires superuser access)
adb shell
su
mount -o remount,rw /system
cp /sdcard/<NamefromOpenSSLOutput>.0 /system/etc/security/cacerts
chmod 644 /system/etc/security/cacerts/NamefromOpenSSLOutput>.0
chown root:root /system/etc/security/cacerts/NamefromOpenSSLOutput>.0
mount -o remount,ro /system
Then we should reboot the device
reboot
Once restarted if you got to on the Android device (names may vary by version) Setting > Security > Encryption & Credentials > Trusted Credentials > System you should see mitmproxy's certificate listed.
In December 2018, Privacy International released a report highlighting how Facebook can track you via Android apps even when your not a Facebook user. Frederike Kaltheuner and Christopher Weatherhead presented the findings of this report at the 35th Chaos Computer Congress (35C3). Privacy International committed during that talk to releasing their environment used to conduct this research so as allow others to either repeat or expand on it.
Although the tools themselves are relatively trivial and there are already generic toolkits (such as Kali) that encompass them, this Virtual Machine is specifically for one purpose, and includes documentation to assist with that purpose, hopefully negating some of the necessity to read forum posts or disparate documentation.
This is a replica of the original environment used to conduct the research, there are a number of reasons why the original environment was unsuitable for redistribution, principally potential issues around licensing due to the use of proprietary drivers. Additionally the original environment was devised in June 2018 and many of tools and features had been subsequently update, the updated software is present in this environment.
This toolkit is built around a flaw that exists in the trust paradigm used extensively on the Internet. When secure connections are established such as HTTPS, the client checks against an internal store of "trust anchors" in its "trust store" known as certificate authorities (or CAs for short). CA's exist in most operating systems through a number of methods, predominantly commercial agreements. This toolkit introduces a CA that we add to the "trust store" (see step 5 and 6 above) which allows us to intercept secure traffic in transit, because the client now trusts this CA in addition to the preconfigured ones.
This is where the term man-in-the-middle comes from, as we are sitting in the middle between the client and the server, intercepting and analysing the communication.
Inside this toolkit there are number of components which work together to allow for interception of data in transit. Let us talk through each element and how they work together, then we will discuss the various ways that they can be used
Virtualbox is a free cross-platform Virtual Machine manager, it allows for one operating system to be run inside another by emulating the features of a physical computer virtually. As far as the operating system inside the Virtual Machine is concerned it is running on physical hardware (mostly). The advantage of doing this is that the hardware is abstract and software portable, so can be redistributed, it also means that the activity inside the virtual machine is contained and isolated from the real operating system that you use.
Debian is a flavour of linux, a unix-like kernel and operating system architecture. Debian Buster is currently the development build for the next major release of the operating system. Buster differs from the previous release (Stretch) in that it incorporates an updated version of Python which eases the installation of mitmproxy, a major component of this toolset
mitmproxy is an open-source proxy (intermediary) server written in Python, it essentially takes connections in on one side, opens them up for analysis and then forwards them on to the other side in full-duplex. Its key feature is that it generates the required certificates in real time so that the incoming connection (from the client), believes that the proxy is the real destination. In the configured environment mitmproxy is configured in "transparent" mode, this means no additional settings need to be changed on the client other than the installation of mitmproxy's certificate.
dnsmasq is a lightweight DNS (domain name service) and DHCP (dynamic-host configuration protocol) server, it serves two purposes, firstly it gives out IP addresses that allow devices to join a network without having them be configured manually this includes setting all the parameters such as DNS server, default route and domain. Secondarily it services DNS requests which involves predominantly turning domain names such as privacyinternational.org into IP addresses such as 144.76.205.68
hostapd is a configurable 802.11 (Wireless LAN) daemon, it allows wireless devices to be configured in a number of ways. In this toolset it is disabled by default as we don't know what WLAN controller you are using, it does however have a sane default configuration setup and can be easily enabled.
iptables is the de-facto standard firewalling system in many linux based operation systems, it uses a table of human-readable rules to categorise and manipulate traffic through a set of well known networking states. In this toolset it serves two purposes, it feeds traffic from ports 80 (http) and 443 (https) into mitmproxy, it also allows for other connections to be masqueraded so that the VM shall act like a router.
+----------+ +-------------------------+ +--------------+
| Public +----------+ Privacy International +-------------+Android Device|
| Internet | || MitmProxy Environment || Wireless: |with mitm cert|
+----------+ || VirtualBox ||pi_mitmproxy+--------------+
enp0s3+---------------------------+USB WiFi
(NAT) Dongle
Lets walk through the diagram from left-to-right. The Internet as displayed in the diagram, would be which ever way you usually connect to the internet. Adapter 1 in virtualbox (enp0s3 inside the VM) should be set as a NAT device, this will mean that the VM will use the hosts network connection to gain access to the internet.
The Virtualbox VM should be booted and mitmproxy should be running before trying to connect any downstream devices. There are scripts on the desktop which are explained in more detail in the usage section.
To the right of the VM is a wireless NIC (we assume a USB dongle), in the shipped configuration this should be disabled, and should be configured before attempting to run mitmproxy. It will host the Wireless network pi_mitmproxy
Finally on the far right is the device you want to analyse, this is assumed to be an Android Phone. The following changes will need to be made to device
- It will be connected to the network willthat provided by the WLAN NIC (
pi_mitmproxy) - You will need to install mitmproxy's certificate into the devices trust store, further instructions for this can be found below
- A Notebook or Desktop, with internet access, capable of running VirtualBox 6.x
- A Device you wish to analyse, in this guide an Android phone is assumed, and it is assumed to be rooted
- A wireless USB Dongle, the main guide will assume a Ralink based one, installation and configuration of the multitude of wireless network adapters falls outside the scope of this guide
Our suggestion for USB Dongles is the following:
amzn.com/B019XUDHFC (they are not fancy, but they do work) (search amazon for ralink rt5370 in your own country)
As your environment and hardware will be different to the one used by Privacy International, there are few things worth checking before starting the Virtual Machine.
The number of assigned CPU's and quantity of Memory
You may want to check that you can allocate the required resources for the VM, in its default state the VM is allocated 2CPU's and 1024MB's RAM. This means that the minimum requirements are 2CPUs and 2048MBs RAM on the host. This setup won't be particularly performant and we strongly recommend increasing both of these to 4CPUs and 2048MBs RAM if your host system has the capacity
Network Controllers
You should check that your Wireless NIC is correctly connected to the Virtual Machine, its best to do this before trying to boot the VM!
Internet Connectivity
You can test this after the VM has booted by opening a browser and seeing if sites like privacyinternational.org load correctly.
Desktop Items
There are a number of desktop items, most of them linking to configuration files. Below is a list of the filenames and their purposes
- LICENSE.md - GPL 3.0 Licence
- README.md - This Readme
- administer_service.sh - A graphical user interface for managing systemd services
- change_dhcp_settings.sh - An editor for `/etc/dnsmasq.conf`
- change_network_interfaces.sh - An editor for `/etc/network/interfaces`
- change_sources_list.sh - An editor for `/etc/apt/sources.list`
- change_wificonfig.sh - An editor for `/etc/hostapd/hostapd.conf`
- mitmproxy_start.sh - A script to start mitmproxy (mitmweb)
- mitmproxy_stop.sh - A script to terminate (all) mitmproxy's
- change_wireless_interface.sh - A script to configure wireless interfaces automatically
Note: We would STRONGLY recommend clicking the "Execute in Terminal" option when executing any of these scripts
In the browser (Firefox)
In the browser there are three bookmarks
- Manual - This Document (Rendered)
- MitMProxy - A link to the local mitmweb instance, frontend for mitmproxy
- Jeroenhd Excellent Guide to installing Certificates in Android Nougat and late
- mitmproxy Documentation - A link to the official documentation for mitmproxy
Workflow
-
Once the VM is started we suggest reading this manual in full!
-
Run
change_wireless_interface.shfrom the Desktop and insure that your wireless network interface is correctly configured. -
Enable
hostapdby running the following command in a terminalsudo service systemctl enable hostapd -
We then recommend restarting the Virtual Machine. If you are still in terminal window, you can do this with the following command:
sudo shutdown -r now -
Once the machine is restarted you should be ready to do analysis. To do this, start mitmproxy using the
mitmproxy_start.shscript on the desktop -
Once you have captured data, you can use
mitmproxy_stop.shto end the mitmproxy process. Your data will be saved in aflow-*.capfile in/home/privacy/
Google changed the way the certificate store works in Android Nougat, please follow this guide to correctly to install mitmproxy's certificate. If you are using earlier versions of Android, you should be able to visit mitm.it once connected to your mitmproxy access point, and install the certificate directly. The abridged steps for installing on Nougat and later are below.
The abridged steps are as follows: Get the ID of the Certificate in a format that Android Expects
sudo openssl x509 -inform PEM -subject_hash_old -in /root/.mitmproxy/mitmproxy-ca-cert.pem | head -1
Note the output: e.g "abcdef12"
Copy the Certificate to the Device over ADB
adb push /root/.mitmproxy/mitmproxy-ca-cert.pem /sdcard/<NamefromOpenSSLOutput>.0
Remount the /system partition, copy the certificate, set its permissions (requires superuser access)
adb shell
su
mount -o remount,rw /system
cp /sdcard/<NamefromOpenSSLOutput>.0 /system/etc/security/cacerts
chmod 644 /system/etc/security/cacerts/NamefromOpenSSLOutput>.0
chown root:root /system/etc/security/cacerts/NamefromOpenSSLOutput>.0
mount -o remount,ro /system
Then we should reboot the device
reboot
Once restarted if you got to on the Android device (names may vary by version) Setting > Security > Encryption & Credentials > Trusted Credentials > System you should see mitmproxy's certificate listed.
The methodology that Privacy International used to do our analyses is as follows:
All data being transmitted between Facebook and apps is encrypted in transit using Transport Layer Security (TLS, formally SSL). Our analysis consisted of capturing and decrypting data in transit between our own device and Facebook’s servers (so called "man-in-the-middle") using the free and open source software tool called "mitmproxy", an interactive HTTPS proxy. Mitmproxy works by decrypting and encrypting packets on the fly by masquerading as a remote secure endpoint (in this case Facebook). In order to make this work, we added mitmproxy's public key to our device as a trusted authority. The data exists on our local network at time of decryption.
The Android handset used was running a "pure" version of Android and connected to a new Google Account setup for the sole purpose of this research. Once the device setup with appropriate setting (pertaining to Wi-Fi, certificate trust, security such as PIN and screen lockout, developer tools such as showing touches) a full phone "nandroid" backup was taken so the device could quickly be returned to a known state. This is particularly important as considering that when some apps are installed and run, they continue to run in the background potentially polluting the results.
After each wipe the following steps where undertaken:
- Connect to a non-intercepting Wi-Fi
- Download the Application from the Google Play Store
- Connect to mitmproxy VM (via Wi-Fi), and create a new flow
- Start Screen Recording using ADB, start screen recording in Virtualbox
- Open the app, and do various activities for up to 320 seconds (if the app requires sign up to use, Google account created at the start of the process)
- Save screen recording off the phone and stop the flow in mitmproxy
- Reboot to recovery and restore the nandroid backup, ready to restart the process
- Reboot the device
Details of the analysis were recorded using the following methods:
- All session data that traversed mitmproxy ("flows") where recorded and stored, so they could be analyzed further, and shared later should the need arise. (in the environment this happens automatically using the scripts provided)
- All activities and interactions on the screen where recorded as video using Androids Developer Bridge's (ADB) screenrecord. (you may wish to do this for your own documentation purposes)
Common issues and their fixes
Virtualbox installation note for MacOS users
[MacOS]: If you are using High Sierra or later, you will need to unblock the kernel driver that Virtualbox installs. This can be done in Security & Privacy in the System Preferences, a prompt should appear during installation.
Starting the environment
I'm having issues starting the environment
If you are having issues starting the environment, check that Virtualbox is installed correctly (see note above). Check that Virtualbox is at least version 6.x.
If you are still having problems, create an issue on the github tracker.
I'm having issues with display
The environment was compiled with Virtualbox guest additions, which means that the display should automatically resize. We suggest using the View -> Virtual Display menu, and setting the virtual display to at least 1920x1080
The environment is slow or unresponsive
You may want to check that you can allocate the required resources for the VM, in its default state the VM is allocated 2CPU's and 1024MB's RAM. This means that the minimum requirements are 2CPUs and 2048MBs RAM on the host. This setup won't be particularly performant we strongly recommend increasing both of these to 4CPUs and 2048MBs RAM if your host system has the capacity
The Firefox window that opens states "Internal Server Error"
The virtual machine didn't have internet access at startup, this is a non critical error, it just means that the documentation won't be displayed in and aesthetically pleasing way, it can still be opened by using leafpad (or any other text editor) with the README.md on the desktop.
Using the tools
My wireless USB isn't appearing in iw list and its definitely connected to the VM
The environment only has drivers/firmware for Ralink adapters, other adapters (such as Realtek) will require their own drivers to be installed, configuring your specific wireless nic falls outside the scope of this document. Either Google <WLANChipset> Debian or contact your vendor.
I can't see a wireless network called pi_mitmproxy
Check that hostapd is running, that the nic is correctly set in the /etc/hostapd/hostapd.conf configuration files. You can always start hostapd in debug mode using this command:
sudo service hostapd stop; sudo killall hostapd; sleep 4;sudo hostapd -p /etc/hostapd/hostapd.conf
My device is looping between connecting and disabled, when I look at the "Wi-Fi" setting in Android
This is indicative of dnsmasq not correctly issuing an IP address. You can restart dnsmasq:
sudo service dnsmasq restart
It may also be because the wireless nic is not correctly defined in /etc/network/interfaces/ check that the nic has a valid stanza in that configuration file, then restart networking and dnsmasq
sudo service networking restart; sudo service dnsmasq restart
Note DNSMasq occasionally experiences a race condition with
hostapd, restarting both services should rectify this.sudo service networking restart; sudo service hostapd restart; sudo service dnsmasq restart
My device is connected, but states it has no internet access
This means either mitmproxy isn't running, or you legitimately have no Internet. If mitmproxy is running, check that you have internet. You can do this by trying to ping google.com in terminal, if you are seeing issues/timeouts, reconnect and restart the VM
mitmproxy says Client Handshake failed. The client may not trust the proxy's certificate
The certificate has not been installed on your handset correctly, please see: Android Nougat or Later Certificate installation
mitmproxy says Initiating HTTP/2 connections with prior knowledge are currently not supported
This is a known issue with mitmproxy and recent versions of Android, the IP's being connected to appear to belong to Google. See this Github issue for more detail
I'm using Android Pie (9) or later, and I'm having difficulties
You may experience issues with Android Pie, Android Pie supports TLS 1.3 which has mitigations against man-in-the-middle, some providers (like Facebook) won't negotiate a connection if there is a man-in-the-middle detected
I'm trying to connect my device for ADB, but Virtualbox won't recognise it
Modern phones that use USB-c type connectors may not have backwards support for USB 1.1 that ships by default in Virtualbox, you may need to install the Virtualbox extension pack (if you are able to comply with the licence), to and configure USB 3.0 in the VM settings.
My device is connected, but I can't adb shell
Make sure that USB Debugging is enabled (this may involve enabling developer mode on your device). If USB Debugging is enabled, check that your device is connect in "File Transfer", "MTP" or "PTP" mode (some devices don't allow the bridge to function in "USB Charge only" mode)
In adb shell command su is not found
Your device hasn't been "rooted", make sure your device is "rooted" by searching for appropriate documentation online. PI accepts no liability for a device becoming inoperable or data loss due to an incorrect rooting process.
Privacy International would like to thank all the contributors to the Open Source projects used in this environment.
Privacy International would also like to thank:
- The 35th Chaos Congress (35C3) and Chaos Computer Club (CCC) - for their help and support.
- Exodus Privacy- For their original research and tools
- Mobilsicher.de - For their research
This document was contributed to by:
From Privacy International:
Christopher Weatherhead, Frederike Kaltheuner, Eliot Bendinelli, Ed Geraghty and Eva Blum-Dumontet
Further pull requests will be accepted
Privacy International is committed to fighting for the right to privacy across the world - but we need your support.
Individual donations are incredibly important to us, and allow us to fund work like this
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent