netsec-ethz / scionlab Goto Github PK

See https://github.com/netsec-ethz/scionlab/wiki/Data-migration

Depends on #78

Provide a way to perform automated end-to-end tests

Add an admin action to promote/demote an AS.

• Update TRC
• Update affected AS certificates -- deal with issuing AS removed from the core!
• Need to determine how to best handle AS links (e.g. remove all or automatically promote/demote).

Create a Form: prompting for:

• attachment point
• label (optional)
• VM?
• VPN? (if available on the selected attachment point?)
• if not VPN:
  • IP
  • NAT: internal IP, internal port (optional)
• port

Create a View to create a new user AS using this form. On submit, this will:

• generate/assign an AS-identifier
• generate keys and certificates for this AS
• create an AS object in the DB with:
  • ISD set to the ISD of the attachment point
  • exactly one Host object, with IP=localhost
  • one Service object for each of the "normal" AS services (ZK, BS, CS, PS)
  • a Link to the attachment point
  • VPN, if configured
• (later: trigger updates for attachment point)
• redirect to the AS detail page

Create a View to see the details of and edit an existing AS, using this form. On submit, this will:

• update certificates if attachment point changed to different ISD
• update the properties (as above)
• redirect to the same page

Not part of this issue: trigger configuration update for attachment point

Add support for (privileged) users sending an invitation (by email) to potential new users. The email should contain a link to a pre-filled registration form.

We can hopefully make use of either of the following libraries:

• https://github.com/bee-keeper/django-invitations (for invitations only)
• https://github.com/bennylope/django-organizations/ (for invitations and a concept of organisations, which we may want to use for quota management)

Currently, when creating a UserAS, the Host running the corresponding border router interface on the attachment point is selected using the function AttachmentPoint.get_host_for_useras_interface which simply makes a sensible choice.
If we have a setup with an attachment point AS running on many different hosts, we may have to make this configurable.

The end-to-end tests will be great to determine whether the generated configuration actually works, but it will still be useful to have tests that verify that the generated configuration is identical to a known baseline (for faster feedback during development and to narrow down the cause of broken end-to-end tests). Maybe, we can simply check-in entire config tar.gz files and compare against those.

Most templates have been created without styling, forms are rendered plainly using form.as_p.
Try to use django-crispy-forms for form rendering.

• registration/activation forms
• login/logout and password change forms
• user_as_add/user_as_details: form already looks decent, consider using crispy to simplify the code

Setup the continuous integration system & provide guidelines for adding tests. This should cover

• unit tests (standard python unittest)
• django tests
• webtest and/or selenium

Check that current configuration can be adequately represented in new data model. Determine the required steps to import the data into the new coordinator & create issues.

Infrastructure:

• SCION config
• VPNs
  • Note: we don't need to migrate the configuration of the point-to-point VPNs used e.g. to connect to GEANT into the data model.

Coordinator:

• User meta data
• User ASes
• ?

All the TRCs and Certificates have an expiration timestamp. Currently, this is just set "far enough" in the future to avoid having to deal with this. With all the functionality to update and re-issue certificates we already have in place, it is should be easy to set a mechanism that updates certificates before they expire.
We should be able to use recurring tasks from huey for this (adding huey is already on the way).

Add an admin action to check the current network topology configuration for problems.

• ASes have all the required services configured
• All ASes connected to it's ISD cores
• No port clashes
• etc.

Note: related to #46; all the checks proposed there as validation can also be double-checked here.

Plan and validate and implement the solution.
The requirements are:

• User ASes running automated updates will continue working seamlessly.
• There will be a script (or similar tool) to update dedicated machines running user ASes.
• Both VPN and direct connection must be supported.
• The ASes are authenticated via account_id and account_secret at least the first time they contact the new Coordinator.

May require some model changes as well.

• Add an API to obtain the full configuration for a host, as a tar.gz-ball, containing

  • generate the updated gen/ folder for the attachment point
  • generate the updated VPN server configuration

  The parameters for the API are

  • an identifier & a secret for the Host (assume we'll use https)
  • the version of the configuration previously deployed, if any

  The response is

  • 304 not changed, if the config version supplied is already the latest version
  • 200 and a tar ball, if the config has changed
  • 204 no content, if the config fpr the host is empty

• Add a script/binary (sh, python, go?) to run on the host, which will query this API, unpack the config tar-ball and restart services as required.

  For user-ases, running this script may be used as alternative to manually downloading and unpacking the tar ball.

  For "managed" Hosts, this update-config script is run either

  • triggered on the host by a timer or similar event
  • triggered by the scionlab server

  Note: might be able to re-use the script created created in netsec-ethz/scion-coord#299

• Add a functionality to the scionlab server to trigger this config deployment on managed hosts. Either using ansible or plain ssh.
  This triggering should

  • asynchronous
  • rate limited ?
  • avoid multiple simultaneous processes for the same target host

Add a function to generate the configuration tarball for a user AS. This should include:

• a README, either for dedicated or VM setups
• the gen/-folder, generated for the (unique) host of a user AS (see #8)
• VPN config (see #10)
• Vagrantfile, if VM setup

Add a new function create_gen(host, directory)
For one Host object, generate the configuration for the gen-folder, by traversing all the related Interface and Service objects.
Always write a configuration for sciond.

After every change to a user AS, the configuration of the related attachment points (two APs involved if the user changes the attachment point) needs to be updated.

• The deployment of the configuration to the AP should be triggered immediately, but with a limited rate (e.g. at most once per minute)
• For each host, only deployment process may be active at any given time
• The deployment should obviously be run asynchronously (with whatever django/python mechanism is most suitable)

Note: If required (e.g. due to VPN-IP-assignment issues) we can re-implement the "updates pending" status, to disallow further changes to the user AS before the deployment has been completed. For now, I'd like to be optimistic that we can solve these issues without such a status.

Depends on #7 #13

For the infrastructure ASes we can support using IPv4/6 by (manually) setting a per-Interface public_ip. For the APs, we'd have to automatically choose which address to configure, based on whether the user configured a IPv6 address. Currently, there is no field in the model where the options for IPv4/6 addresses could be stored; a Host has only one default public_ip.

Add back support for image builder.

Low priority. Image builder might be entirely replaced by having binary packages.

In the admin pages, creating/changing links can create an invalid SCION network topology.

We should implement proper validation to avoid creating impossible configurations:

• no PARENT/CHILD loops
• CORE links only between core ASes
• no PARENT links for core ASes
• no PARENT/CHILD links outside of ISDs

Other things that can be validated (less important):

• multiple VPNServers on same host must use different subnet
• multiple VPNClients on same host must use different IP

Customize the ModelAdmin classes for the scionlab-model objects, to quickly get a somewhat workable admin interface to configure the infrastructure ASes (i.e. a replacement for scion-web).

Add a function to generate the VPN client configuration for a Host: write_vpn_client_config(host).
Add a function to generate the VPN server configuration for a ManagedHost: write_vpn_server_config(host)

Note: this should support multiple VPN clients on the same host.

Import Users from scion-coord data dump.

See https://github.com/netsec-ethz/scionlab/wiki/Data-migration

Depends on/connected with #75

Use django-recaptcha to add a captcha to the registration view.

The integration of the huey for the config deployment tasks is fairly fiddly. We should have unittests that only test the actual fiddling with huey.

Somewhat tricky to set this up for testing because the huey consumer needs to run in a separate process.

Create the main page for a user: this contains

• a friendly greeting
• a list of the user's ASes with "edit" links to the detail pages
• a button to create a new AS, enabled only if quota not exceeded

Gather status info from attachment points / core-ASes.
This should then enable to automatically disable UserASes that have not been active for some time.

Adapt the default user model to:

• use email instead of a username for login etc.
• only ask for email during registration
• add an organisation field (probably not: we might be able to use e.g. https://github.com/bennylope/django-organizations/ if we want to do this; see also #21)

Adapt the django_registration.forms.RegistrationForm and the corresponding template (depends #3).

See notes on extending django user model:

• https://stackoverflow.com/questions/37332190/django-login-with-email/43209322#43209322
• https://www.fomfus.com/articles/how-to-use-email-as-username-for-django-authentication-removing-the-username

Configure PostgreSQL instead of sqlite as the DB backend, at least in the production configuration.

Need to ensure that IP addresses given by user are safe to use in the attachment point configuration.

• Not localhost / ip6-localhost etc.
• No broadcast addresses
• other?

The UserAS creation/edit form needs a little bit of JavaScript logic for the following dynamic behavior:

• VPN option should only be shown/enabled if the selected attachment point supports VPN
• Public IP/bind IP settings should be hidden if VPN is enabled
• The bind-address settings should be hidden/collapsed ("My host is behind NAT")

Related: #31

The file under gen/overlay is missing when generating the configuration. Without it, at least the current BS refuses to start.

Use the "two-phase" registration model from django-registration: https://django-registration.readthedocs.io/en/3.0/

• Install and configure the django-registration module as described here:
  https://django-registration.readthedocs.io/en/3.0/quickstart.html#configuring-the-two-step-activation-workflow
  • Add "django_registration" to the INSTALLED_APPS list
  • Configure the URLs
  • Define these 6 templates (minimal working version, no styling):
    https://django-registration.readthedocs.io/en/3.0/quickstart.html#default-templates
• Customize the RegistrationForm to additionally include
  • First Name
  • Last Name
  • and organisation
  • accept terms of use (see django_registration.forms.RegistrationFormTermsOfService

Create a nice landing page.

• Introduction
  • What is SCION
  • What is SCIONLab
  • What is a (SCION-)AS
  • -> Register to run your own ASes today 😉
• Pictures
• Login/register buttons/form

Add/populate navbar, hamburger menu with "my ASes", "logout" and "password change" and link to scion-architecture, tutorials etc.

Currently implemented by a simply delaying for one minute before actually running the update. All updates during this time will be bundled together. This introduces unnecessary delay.

As suggested by netsec-ethz/scion-coord#231

Implementation:

• require acceptance on registration, using django_registration.forms.RegistrationFormTermsOfService
• include a "Terms" link in the footer

We should use a vagrant box with prebuilt SCION binaries, as in @juagargi's netsec-ethz/scion-coord#262.

As first step of configuration deployment, the AS certificates and TRC need to be generated if marked as needs_update.

The models contain options to configure a bandwidth-test-server (Service.BW) and pingpong-server (Service.PP). During the generation of the gen/ folder, these are currently ignored in the code.

Add a supervisor config file to start these services.
A starting point is _create_gen in generate.py:

Currently have (manually configured) servers that use UDP/TCP setup; two openvpn servers one UDP, one TCP, both handing using the same address range for VPN addresses, using an address-learn-script to explicitly set routing rules per client.
We'll need this, or something similar again to have "no UDP" fallback.

Create a script that creates the data model representation for the existing SCIONLab infrastructure topology (somewhat analogous to the current fixtures/testtopo.py).

This should:

• create the AS,es Hosts, Routers/Interfaces/Links, Services (etc)
• import keys/certificates

The idea is to start using the new coordinator to start configuring (some of) the scionlab infrastructure hosts.
The attachment points will not yet be included, but we could still create them as "dummy" entries.
(Let's not bother with the VPN settings yet).

Many home users have public IP addresses but those are dynamically changing (e.g. users connected over DSL or LTE). Those users could be supported by relying on a static hostname (such as one assigned through dyndns) instead of VPN, thereby avoiding performance degradation.

Profiling of the DB models and queries. Can be based on the existing tests. Preferrably also using PostgreSQL.
The goal is to detect big performance problems early and make the necessary adjustments to the data model.

Use django.contrib.auth to implement login, logout, password change and reset functionality.

https://docs.djangoproject.com/en/2.1/topics/auth/default/#using-the-django-authentication-system

• Configure the URLs for the contrib.auth-views, analogous to https://github.com/django/django/blob/master/django/contrib/auth/urls.py
• Create basic templates for these 8 views and set them in the URL config

scion-coord hashes passwords using golang.org/x/crypto/scrypt.

https://github.com/netsec-ethz/scion-coord/blob/888326037e3202d5622686438dec1e035a6b0c73/models/user.go#L83-L85

Django does not support this hash algorithm out of the box. We may be able to use django-scrypt, but it appears to be a bit outdated -- we might have to update or re-implement parts of it.

Add gen/ia file, containing the ISD-AS (no trailing newline) to the config tarball, like the current coordinator does.
Not having this will break web-app and various other scripts/tools/apps that introspect the gen folder.

Add a button to the AS detail view to download the configuration tarball.
This should just generate the tarball and then serve it. If there is a simple way to cache the tarball to disk using django's caching framework, then we should enable that.

Create script to compare generated configuration tar-ball with existing gen/-folders.

• Sort topology.json, yaml files etc.
• toml files?
• supervisor files will have differences, maybe just ignore them

When a user registers, she will currently just be redirected to login, without any indication that an email has been sent to activate the account.