iclab / centinel Goto Github PK

Add the ability to scan from VPNs by registering VPN config files as clients

fix bug in backend where we try to call a json dict. from execution on one of the Pis:

Traceback (most recent call last):
File "/home/pi/centinel/centinel.py", line 36, in
centinel.backend.sync(configuration.params)
File "/home/pi/centinel/centinel/backend.py", line 137, in sync
for experiment in (set(user.experiments) - available_experiments):
File "/home/pi/centinel/centinel/backend.py", line 37, in experiments
return self.request("experiments")["experiments"]
File "/home/pi/centinel/centinel/backend.py", line 29, in request
return req.json()
TypeError: 'dict' object is not callable

We need an SSL certificate for ICLab to secure communication between the client and the server. We need to either buy a cert or use self signed certificates.

@rpanah, can you look more into this?

per Abbas, Phillipa, and my discussion today, we will update the DNS primitive to do the following:

1. add the ability to send chaos queries
2. ensure that we wait for a potential second response (just read from the socket after some time period)
3. use dnspython to handle DNS packet construction
4. add the ability to send multiple DNS requests at once (ensure that each of these uses a different port to distinguish responses)

Create a page that shows centinel's deployed client base. This will be similar to networkdashboard.org.

Questions: @rpanah, @gsathya

• Is this visible to contributors or everyone?
• How good is good enough for geographic aggregation?
• Are we giving out too much info if we say whether the node is a VPN or not? This is important from a network measurement perspective, but it may identify users of the tool.

This dashboard functionality should

• aggregate location info to the country level to prevent PII leakage

In the event of a single test causing an exception, execution is interrupted and the output for all of the tests that were run before it will not be written to output.
There better be an exception handler when iterating over tests to avoid loss of results.

Per the discussion on issue #16, we should refactor the config to be a dict of dicts (stored in json format) rather than a static Python file.

One way we can go further is that a lot of implementations we see an HTTP get as done through a Python http library not a browser. Writing the same GET request through a headless webkit browser would be more realistic representation of a users browser. Ie - instead of just getting an HTML source of the front page it would do all the GETs of content of the page - js,images, etc.

Getting PCAPs of this kind of process would give you much richer data that is more in line with the user experience.

Add unit tests for each experiment

Submit registration that takes advantage of the new functionality added

We are now shipping a copy of our certificate bundle (the validation chain) with our client because we ran into issue verifying the GoDaddy certificate from certain machines.

The error we get from Python requests is:
ben@ben:~$ python
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import requests
r = requests.get("https://server.iclab.org:8082/geolocation")
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
return request('get', url, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
resp = self.send(prep, *_send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
r = adapter.send(request, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 382, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Before we deploy to our initial set of clients, we need to make sure that we are logging all of the exceptions. This will allow us to understand what went wrong when/if our program crashes

use a config file in ~/.centinel.conf to store the path or give a command line argument such as
$ centinel -i /path/to/input_files

audit the roles for the centinel DB to see if we need to add any new roles and to ensure that clients can only access their own data

Ensure that there are not any problems encoding data into the JSON format. If there are problems, Base64 encode potentially problematic data or switch to a new encoding type

Checks include checking for unicode support and ensuring that we can handle raw data like pcaps.

Skype accessibility. The two pieces here -

• Accessibility of the binary of Skype. ie - Can you download a copy and does that hash match what you would get in a given control country? and
• Can you access the chat services in general? A case of blocking of this type has been seen previously in UAE (though not currently) http://www.thenational.ae/news/uae-news/phone-call-revolution-as-skype-is-officially-unblocked-in-uae

Right now, our platform test against all targets every time it is run. This is problematic because a) we want to be mindful of bandwidth constraints and so we need to either probabilistically select targets from a set or use a very small set of targets and b) we will be less fingerprintable if we don't test our entire URL list every time.

On package installation, we are currently installing all available experiments. This is a problem because the experiments we push with the package will be run every time that Centinel is run.

To fix this problem, we need to consider either refactoring centinel.py to not sync with the buitin experiments directory, or modify the package not to include the experiments directory. Perhaps we could add a command line parameter for whether we should check that directory? (this may be useful for testing)

Is there value in shipping a reachability test that runs every time centinel runs (it will not be possible to run centinel without the test if we choose to ship that way)

had problems installing pip package on centinel.noise.gatech.edu. Did i forget to update the package number?

We need to either update the build script or create an option for the Centinel binary to install the crontab entries for the user. @gsathya, do you have a better, platform independent solution?

@rpanah, @phillipa, @feamster, @gsathya, philosophically, do we want the user to opt-in or opt-out of our platform? I'm inclined to think that we should force users to opt-out of uploading results and provide plenty of documentation/make it easy to opt-out (maybe even provide a command line flag). Is anyone strongly opposed to an opt-out model?

Is it possible to get it through centinel? how would it be done? Ethiopia, China, and other places implement censorship via reset packets and having this available would be a big plus.

get the user to install paris traceroute, then use that as a measurement primitive (we are now assuming that centinel will/can be run as root)

The goal of this issue is to daemonize/ get Centinel running consistently in the background on a machine.

For now, we will only support Linux systems so we will use the cron.hourly and cron.daily locations to setup a) autoupdates and b) setup a schedule for syncing with the server and running measurements.

I added a variable to config.py which is not committed (config.py)

Put a cron job on someone's machine to build/upload the pip package every night using the master branch. This will create a nightly dev build.

Create a command line argument that will walk the user through doing the informed consent.

https://github.com/iclab/centinel/blob/master/centinel/backend.py#L142 should return boolean not None

Add the URLs from the citizen lab into experiments per https://github.com/citizenlab/test-lists

Hi guys,

I just tried installing and running centinel. It seems to basically do what it did when we left off in May (modulo not crashing on Turkey now).

Is there an ETA on when documentation/a set up script to put up a new node that is part of the larger platform will be available? CitLab folks are gone after Friday for a couple of months.

-Phillipa

in response to the cascading failure (tons of centinel clients continuously syncing), we need to create controls to prevent such a failure in the future.

Some partners we have are interested in experiments that determine site uptime, RTT mainly to look at DDoS behaviour. On the July 9th Presidential elections in Indonesia for instance there is a strong suspicion (given previous elections) that opposition sites will be DDoS'ed. Measuring the extent of this would be useful.

Currently, we upload each results file using an individual HTTP connection. This is fine in many cases, but if there are a lot of results to upload, then this is really inefficient/slow. We should find a way to consolidate the results together before uploading/downloading them.

If the .centinel directory does not exist for the server, it gives very obtuse messages like "sqlalchemy.exc.OperationalError: (OperationalError) unable to open database file None None". The config file should create the directory if it doesn't exist

While testing the VPN code, I discovered that the Python logging module will keep logging to the original file if you make multiple calls to logging.BasicConfig. It would be nice to address this because currently, all of the VPN logs go into 1 file.

@gsathya, let me know what you think.

update /clients to enumerate clients and their properties, not just usernames

Determining SSH accessibility/throttling such as in Iran last election:
https://twitter.com/CDA/status/331523695107514368

This issue may not require any code, but we need to consider if we can design the experiments in a better way to encourage extensiblity. They may be fine, in which case we should close the issue, but either way we should have discussion here.