danwent / perspectives-server Goto Github PK

We should be able to easily compress the XML before sending it to the client. This could cut bandwidth usage, possibly allowing notaries to serve even more clients.

We can store the final compressed calculated data as the payload inside the cache.

We'll need to test this to see if uncompressing data has any performance impact on the client, and if that tradeoff is worth doing.

Commit 325367f introduced a change in admin/start_webserver.sh and as a result the main process no longer detaches itself from active console.

The original Perspectives paper includes the feature of validity checks and data redundancy to "[limit] the power of a compromised or otherwise malicious notary server"[1]. In this setup, each notary acts as a 'shadow server' for several other notaries, storing a record of their observations. We could implement this to help guard against data changes or corruption.

[1] Section 5: "Detecting Malicious Notaries"
http://perspectivessecurity.files.wordpress.com/2011/07/perspectives_usenix08.pdf

When a website legitimately updates its certificate it normally has to wait for the notary's scheduled scanning process to run before information is updated. This is not ideal as it can take a long time for the updated cert information to propagate, and users will see a scary red 'X' for the site.

Add an optional feature that allows any visitor to submit the name of a site to be scanned on demand. The notary will then attempt to scan the site just as it normally would, using the same settings, and update its database with the new certificate information. Legitimate site owners or good samaritans could then notify notaries that a certificate has changed.

It is important that the notary itself do the scanning and that the only input allow by the user is the name of the site. Also, scans should be rate-limited, perhaps by IP address, by target site, and by total requests in 24 hours, so that the notary cannot be overwhelmed.

We should add HTTP Cache-Control headers when we server requests, so that clients will cache and re-use content. This will both help clients get the results more quickly (because they'll already have them) and also reduce load on the server (because there are fewer requests).

What's better than a plaintext query? An encrypted query.
What's better than an encrypted query? A query that never gets sent. It's impossible to crack a request that didn't happen.

Some primers on the header settings:

• https://devcenter.heroku.com/articles/increasing-application-performance-with-http-cache-headers
• https://www.mnot.net/cache_docs/

Can we just use the cherrypy config setting to turn this on, or are we better off adding specific headers ourselves?

It might be useful to set the header expiry to just after the scan process finishes, so the client doesn't have to re-query until the data has changed (though we may have to be careful with client privacy there - set the expiry time in 1-hour chunks, and not down-to-the-second times). But even adding, say, one hour of client caching might help a lot at increasing client responsiveness and reducing server load.

While investigating #34 a long time back I found a number of upgrades and fixes we should make to the scanner code. Creating a ticket to track this. I have a local branch with a lot of code that needs to be cleaned up and properly tested.

Fixes include:

• Extract hex strings to constants to document and explain what they mean (easier to understand and maintain)
• Extract list of advertised ciphers to a more usable form that is maintainable, understandable, and documentable
• Fix SNI scans to use localtime rather than hard-coded timestamp
• Fix SNI scans to use random data on each handshake rather than hardcoded value
• Add more, newer, cipher suites so servers don't reject connections with 'Code 40 Handshake Failure' when they find no acceptable ciphers
• Remove compression support to avoid CRIME exploits
• Fix bug: send TLS 'close_notify' record before closing socket (required by RFC 5246)
• Fix bug: only close scanning socket once
• Move fingerprint calculation to function, so other fingerprint types can easily be calculated on the server's certificate

this will save server CPU for services that are requested multiple times in the same scan period.

SSL Labs' "TLS Deployment Best Practises" doc suggests using a minimum of 2048 bit RSA keys, or ECDSA keys for longer lengths. We should increase notary key sizes.

This will require testing to see if takes noticeably longer to calculate data for sites. We could always implement a background script that calculates site XML after a scan has completed, similar to server version 2.

We do have some hardcode, lads! It's useful most of the time, but let we add something like this :

• --bind-address | -i - An IP address to bind on. Hardcoded 0.0.0.0 is too WIDE =)
• --memcache-host | -mh - An address for memcached
• --memcache-user | -mu - memcached username
• --memcache-password | -mp - memcached password
• --redis-url | -ru - redis url if not default

Just for reference : it's an enhancement, not a critical error

Currently cache duration is controlled by the hard-coded value NotaryHTTPServer.CACHE_EXPIRY. To alter the defaut of 24 hours you have to edit the code. This is not ideal and we should expose the value so it can be passed as an argument on startup.

Currently if a notary scans a site and finds a different certificate than it saw last time, it does the following:

1. Create a new observation record for that site/certificate combination, with a start and end time of the current time
2. Update the previous certificate observation for that site, setting the end time to the current time minus one second

We should not perform the second step - it invents data that we can't vouch for (e.g. the cert could have changed 1 second after we last observed it). It also opens the opportunity for a bad bug: if you don't run your scans regularly, or run scans after a very long absence, the notary would backfill a large time range, claiming the certificate had been seen throughout that time period.

Some hosting services monitor their tenants' activity for things like communicating with known malware or phishing sites. Currently, because Perspectives queries any service (site) that a user visits these queries can generate alerts or flag notary servers as suspicious. Even if a notary server is not running on a hosted site, the owner may wish to prevent some services from being scanned.

We may want to add some way to prevent scans of certain services. I'm a big fan of the idea that you should enumerate goodness rather than enumerating badness (see idea 2). We may want to implement this as a whitelist rather than a blacklist.

A feature like this would also allow us to remove old or broken services if they no longer exist or scans return some kind of persistent failure, saving time by avoiding pointless scans.

Automatically blacklisting sites that fail repeated scans may have security implications. But perhaps there are other things we could do to speed up scan times?

• Use a ThreadPool instead of continually spawning threads? How large should the pool be?
• Only scan one or some subdomains instead of all subdomains for a site? For example, if we scan deviantart.com do we really need to scan any artist.deviantart.com subdomains? Or scan a random sampling each time?

If ssl_scan_sock.py is used to scan sites, SSLAlertExceptions will cause a traceback (see error below). Instead we should catch and print the exception and continue.

Note we'll need to implement catching in all places that call attempt_observation_for_service(). Currently this is three: threaded_scanner.py running scans, notary_http.py running on-demand scans, and ssl_scan_sock.py itself running as the main python module.

In addition, we should decipher the TLS Alert record that caused the error and print it for the user so they can see what went wrong.

Example traceback:

Traceback (most recent call last):
File "notary_http.py", line 359, in run
fp = attempt_observation_for_service(self.sid, self.timeout_sec, self.use_sni)
File "/home/notary/Perspectives-Server/util/ssl_scan_sock.py", line 226, in attempt_observation_for_service
return _run_scan(dns,port,timeout_sec,False)
File "/home/notary/Perspectives-Server/util/ssl_scan_sock.py", line 159, in _run_scan
raise SSLAlertException(rec_data)
SSLAlertException

Currently queries are sent to notaries using unecrypted http traffic. Messages are still cryptographically signed and verified using the notary's public key, so they are guaranteed to be correct and secure, but this is not ideal for privacy. Currently anyone watching the network can see the messages being sent and received.

We should allow clients to send queries over https.

CherryPy currently has an error if the service_id contains unicode, such as 'Bücher.de'.

File "C:\Python27\lib\site-packages\cherrypy_cprequest.py", line 656, in resp
ond
response.body = self.handler()
File "C:\Python27\lib\site-packages\cherrypy\lib\encoding.py", line 188, in __
call__
self.body = self.oldhandler(_args, *_kwargs)
File "C:\Python27\lib\site-packages\cherrypy_cpdispatch.py", line 34, in ca
ll
return self.callable(_self.args, *_self.kwargs)
File "notary_http.py", line 431, in index
return self.get_xml(host, port, service_type)
File "notary_http.py", line 273, in get_xml
service = str(host + ":" + port + "," + service_type)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 0:
ordinal not in range(128)

I'm not sure what the correct action is to take.

having longer sleeps seems to make individual queries take a while, which may lead to more timeouts for on-demand probes.

It doesn't work with the ipv6 sites. There is always "[ No Results ]"

Currently the notary reply does not contain the host name. We should include it so clients can verify that the fingerprint returned is indeed for the host they expected.

Using SSL to encrypt the reply would also protect against attacks, but we should include it in the reply just in case.

Currently util/ssl_scan_sock.py and util/ssl_scan_openssl.py support SNI scanning, but the variable that controls it is hard-coded and set to False. We should expose this as a command line parameter so it can be easily enabled. The argument should remain False by default.

While sensitive data stored in memory can never be truly safe on an untrusted system,
especially in python with immutable strings, garbage collection algorithms, et cetera, we could take steps to enhance our current treatment of sensitive information.

Notary server code could track any strings that contain sensitive information, and overwrite them with e.g. zeros once they are no longer needed or if the notary shuts down. This would be better than doing nothing.

I would like to do the following things:

• Integrate the list_service_ids functionality in threaded_scanner.py
• Start a certain number of ScanThreads (they should get the next address-to-be-scanned themselves from an array(using locking))
• your idea here...

We don't want notary logs to get out of hand. Let's add a log handler to delete logs older than 30 days or a certain filesize so they don't consume an entire disk.

This is built in to python - http://docs.python.org/2/library/logging.handlers.html#rotatingfilehandler

In the spirit of using default values to make it easy for anyone to quickly run a notary, let's default to 30 days and let people specify a different value if they wish.

We should add a switch to allow notaries to use openssl for on-demand scans in addition to using the built-in python scanner. This should be as simple as importing a different attempt_observation_for_service(); from ssl_scan_openssl.py rather than ssl_scan_sock.py.

People setting up their scanning process can of course already run it with the desired python module. This switch would change the behaviour of notary_http.py for on-demand scans.

For analysis of future security incidents, it would be good if we store the full certificate PEM once for each fingerprint seen.

This would be pretty easy to add using a second table that contains two columns: fingerprint, PEM.

Note: we should move certificate data to its own database table ( #30 ) before storing additional data.

Looks like the domain is lost? Why don't we reanimate the project?

Dan Callahan has a good presentation on basic webserver security settings here:
http://pyvideo.org/video/2315/quick-wins-for-better-website-security

We should enable the applicable ones for notary servers. This means:

Render:
1. Enable XSS Protection
2. Set the Content Security Policy
3. Set XFrame options

Adding SSL is covered in #19. Once SSL is enabled we can also add Strict Transport Security.

It may help to anonymize requests and increase user privacy if they can send queries anonymously to notaries. Notaries could accept an encrypted client request and forward it to another notary, returning the response on the second notary's behalf. The request could be encrypted with a public key for the second notary, so the intermediary can't read it.

Like this:

[Client] -------> [Notary A] -------> [Notary B]
..
[Client] <------- [Notary A] <------- [Notary B]

This would help to prevent behaviour tracking and profiling even if notaries are set up by a malicious entity.

Use case: user could host their own notary and use it solely for forwarding requests, to protect their origin.

This should be an optional feature that is disabled by default. To maintain fast performance the notary operator may want to have multiple copies of cherrypy running behind a load balancer, with some of them handling normal requests and some forwarding, etc.

If we implement #26 , forwarding and shadow-monitoring could possibly share the same data-gathering behaviour to retrieve data from other notaries. Forwarding notaries could even keep cached copies of forwarded replies.

How to repeat : try to scan high-secure site, disallowing weak ciphers handshake. For example https://crowdcourse.net

perspectives@crowdcourse:~$ /usr/bin/python notary_http.py --sni --webport 8081 --pycache 100M --cache-expiry 4H --dbtype sqlite --dbname notary.sqlite --echo-screen --dbecho 1
2015-06-19 21:27:50,942 INFO sqlalchemy.engine.base.Engine PRAGMA table_info("t_services")
2015-06-19 21:27:50,944 INFO sqlalchemy.engine.base.Engine ()
2015-06-19 21:27:50,947 INFO sqlalchemy.engine.base.Engine PRAGMA table_info("t_event_types")
2015-06-19 21:27:50,947 INFO sqlalchemy.engine.base.Engine ()
2015-06-19 21:27:50,947 INFO sqlalchemy.engine.base.Engine PRAGMA table_info("t_observations")
2015-06-19 21:27:50,947 INFO sqlalchemy.engine.base.Engine ()
2015-06-19 21:27:50,948 INFO sqlalchemy.engine.base.Engine PRAGMA table_info("t_metrics")
2015-06-19 21:27:50,948 INFO sqlalchemy.engine.base.Engine ()
Using public key
-----BEGIN PUBLIC KEY-----
MIHKMA0GCSqGSIb3DQEBAQUAA4G4ADCBtAKBrAGG5qAoxnLOsoHGG3zJs+W5Yn3+
AOMUzxhZqZsGbIxL46lw1wfgXJQ9MUbdzRjZjpd5TBInbNJbqQiQ0axD01MEgIgf
rCiG+ok7P85iKHdY2vHOnQ12AHDFODGJ/dICSVmLnaCsuoeG66pbNWs9JKTge32I
TJ3dULqzc04hO71j+QVZ3fpkuW49/xXzqz7iPvcGTIosc4S68pffd9dqACXw0KxW
DeYQnjW744sCAwEAAQ==
-----END PUBLIC KEY-----

[19/Jun/2015:21:27:50] ENGINE Listening for SIGHUP.
[19/Jun/2015:21:27:50] ENGINE Listening for SIGTERM.
[19/Jun/2015:21:27:50] ENGINE Listening for SIGUSR1.
[19/Jun/2015:21:27:50] ENGINE Bus STARTING
[19/Jun/2015:21:27:50] ENGINE Started monitor thread 'Autoreloader'.
[19/Jun/2015:21:27:50] ENGINE Started monitor thread '_TimeoutMonitor'.
[19/Jun/2015:21:27:51] ENGINE Serving on 127.0.0.1:8081
[19/Jun/2015:21:27:51] ENGINE Bus STARTED
2015-06-19 21:27:57,143 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)
2015-06-19 21:27:57,143 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)
2015-06-19 21:27:57,145 INFO sqlalchemy.engine.base.Engine SELECT t_services.name AS t_services_name, t_observations."key" AS t_observations_key, t_observations.start AS t_observations_start, t_observations."end" AS t_observations_end
FROM t_services JOIN t_observations ON t_services.service_id = t_observations.service_id
WHERE t_services.name = ?
2015-06-19 21:27:57,145 INFO sqlalchemy.engine.base.Engine SELECT t_services.name AS t_services_name, t_observations."key" AS t_observations_key, t_observations.start AS t_observations_start, t_observations."end" AS t_observations_end
FROM t_services JOIN t_observations ON t_services.service_id = t_observations.service_id
WHERE t_services.name = ?
2015-06-19 21:27:57,146 INFO sqlalchemy.engine.base.Engine ('crowdcourse.net:443,2',)
2015-06-19 21:27:57,146 INFO sqlalchemy.engine.base.Engine ('crowdcourse.net:443,2',)
ERROR:root:Received SSL Alert during SNI scan of crowdcourse.net:443 - 'Fatal (2): Code 40 - Handshake Failure'. Will re-run with non-SNI scan.
Error scanning 'crowdcourse.net:443,2' - Fatal (2): Code 40 - Handshake Failure
^C[19/Jun/2015:21:30:42] ENGINE Keyboard Interrupt: shutting down bus
INFO:cherrypy.error:[19/Jun/2015:21:30:42] ENGINE Keyboard Interrupt: shutting down bus
[19/Jun/2015:21:30:42] ENGINE Bus STOPPING
INFO:cherrypy.error:[19/Jun/2015:21:30:42] ENGINE Bus STOPPING
[19/Jun/2015:21:30:42] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 8081)) shut down
INFO:cherrypy.error:[19/Jun/2015:21:30:42] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 8081)) shut down
[19/Jun/2015:21:30:42] ENGINE Stopped thread 'Autoreloader'.
INFO:cherrypy.error:[19/Jun/2015:21:30:42] ENGINE Stopped thread 'Autoreloader'.
[19/Jun/2015:21:30:42] ENGINE Stopped thread '_TimeoutMonitor'.
INFO:cherrypy.error:[19/Jun/2015:21:30:42] ENGINE Stopped thread '_TimeoutMonitor'.
[19/Jun/2015:21:30:42] ENGINE Bus STOPPED
INFO:cherrypy.error:[19/Jun/2015:21:30:42] ENGINE Bus STOPPED
[19/Jun/2015:21:30:42] ENGINE Bus EXITING
INFO:cherrypy.error:[19/Jun/2015:21:30:42] ENGINE Bus EXITING
[19/Jun/2015:21:30:42] ENGINE Bus EXITED
INFO:cherrypy.error:[19/Jun/2015:21:30:42] ENGINE Bus EXITED
[19/Jun/2015:21:30:42] ENGINE Waiting for child threads to terminate...
INFO:cherrypy.error:[19/Jun/2015:21:30:42] ENGINE Waiting for child threads to terminate...

after that - the bug :

perspectives@crowdcourse:~$ /usr/bin/python notary_util/list_services.py | grep -c crowdcourse.net
0

The URL was lost!

As mentioned in #19, the suggested easy route for increasing CherryPy performance is to place CherryPy behind another server that is meant to be fast and scalable, such as nginx.

Let's create some nginx config files we can use to quickly and easily deploy on notary servers. We can then implement https (#19) and upgrade to higher notary versions later and separately. Adding nginx will let us make the servers responsive again.

We can probably create one set of configs for the default notaries, which should serve ports 8080 (for backwards compatibility), 80, and eventually 443. Other notaries that do not need to serve 8080 can have a config that only servers 80 and eventually 443.

If you do not scan with SNI, ssl_scan_sock gets a 'protocol version' error from some sites:

python ssl_scan_sock.py howsmyssl.com:443
Error scanning howsmyssl.com:443 - Fatal (2): Code 70 - Protocol Version: The protocol version sent is recognized but not supported.

Perhaps this is happening because we're sending an old client_hello message from an older TLS spec? This should be updated.

I set up a Perspectives-Server on a Raspberry Pi Model B with 512MB RAM and a list with more than 1000 lines failed to import (see https://github.com/danwent/Perspectives-Server/wiki). A comment in the file indicates that this may be due to printing a huge MySQL command.

A workaround is described in the above mentioned article.

if a client saw something other than what the notary reported, let the client request a new "on-demand" probe.

we should do something simple to rate limit the number of on-demand probes allowed, since a client could otherwise use a notary to 'hide' its identify while flooding requests to another site.

We currently duplicate a lot of information storing the certificate fingerprint inside the observations table. We should move our data back to Third Normal Form and move cert info into it's own table.

I've installed 3 Perspectives servers, 2 of them are configured to cache data in memcached.
These 2 servers often return stale data on HTTP requests.
I didn't find anything related to memcache in threaded_scanner.py, so I believe that it does not update memcache which causes server to return stale data.

Currently Perspectives uses MD5 certificate hashes. We should upgrade to SHA1, and at least give the option of displaying the SHA1 hashes. This will need to be rolled out in stages:

• Client is able to read MD5 and SHA1 and handle either
• Servers begin storing SHA1 hashes

The first _read_record() in _run_scan() times out for certain domains. One example is git.cocaine.ninja:443.

I tried debugging it but long hex strings without explanation are beyond my level of hardcore.

Alternatively, is there a way to switch to the OpenSSL scanner without changing code?